Compare commits

...
Author SHA1 Message Date
overtrue 2308d91990 fix(tier): honor force for lifecycle references 2026-08-29 12:41:17 +08:00
hectorandGitHub 346388b63c fix: provide cross-repo token for auto-testing checkout (#6831)
The test workflows checkout the private rustfs/auto-testing repository, but
the default GITHUB_TOKEN only has access to rustfs/rustfs, so every checkout
failed with 'repository ... not found' (nightly runs on 2026-08-28).

Pass secrets.PF_TESTING_GH_TOKEN (the existing cross-repo PAT already used
by the performance workflow) to the auto-testing checkout steps in all three
workflows.
2026-08-29 12:00:19 +08:00
cui fliterandGitHub a56439219f fix(version): do not bump version when HEAD equals latest tag (#6828) 2026-08-29 03:25:05 +00:00
Zhengchao AnandGitHub 0fe41da688 fix(ecstore): document audit/notify KVS divergence and fix auth_token redaction (#6816)
Triages the three divergences backlog#2054 found between the audit and
notify default KVS tables, cross-checked against MinIO upstream
(internal/logger/config.go, internal/config/notify/parse.go):

- webhook: audit's extra batch_size/max_retry/retry_interval/http_timeout
  keys match MinIO's DefaultAuditWebhookKVS byte-for-byte, while notify's
  table matches MinIO's notify DefaultWebhookKVS (which lacks them).
  Intentional, not a copy/paste gap — documented with a doc comment on
  each table instead of changed.
- mqtt: audit's stronger QoS/keep-alive/reconnect defaults have no MinIO
  precedent (MinIO's audit logging has no MQTT target at all), while
  notify's 0/0s/0s defaults match MinIO's DefaultMQTTKVS exactly.
  Documented as an intentional RustFS-original choice, not changed.
- auth_token hidden_if_empty: audit had false, notify had true, with no
  MinIO precedent either way (this KVS version has no per-key hidden
  flag upstream). Fixed audit to true, matching notify and every other
  sensitive key in both files (MQTT_PASSWORD, *_TLS_*). Non-empty tokens
  were already redacted identically on both sides via ends_with("_token")
  pattern matching in config_admin.rs — this only changes how an *unset*
  audit webhook auth_token renders in admin config output (omitted
  instead of shown as an empty value).

Refs rustfs/backlog#2054
2026-08-28 17:34:04 +00:00
0953f7e912 perf(ecstore): optimize bounded small-object GET paths (#6808)
* perf(ecstore): bound mid-size GET decode buffering

Use a single in-flight decoded stripe for the gated mid-size GET path and avoid its outer synchronization mutex while preserving the general codec reader behavior. Add full, partial, degraded, error, and cancellation coverage for the bounded reader.

Co-Authored-By: heihutu <[email protected]>

* perf(ecstore): unify small GET path validation

Co-Authored-By: heihutu <[email protected]>

* fix(ecstore): bound mid-size prefetch and preserve gate metrics

Co-Authored-By: heihutu <[email protected]>

* perf(ecstore): cache small-object read path plan

Co-Authored-By: heihutu <[email protected]>

* perf(ecstore): cache GET path plan and verify wiring

Co-Authored-By: heihutu <[email protected]>

* test(ecstore): remove redundant metadata clone

Co-Authored-By: heihutu <[email protected]>

* test(ecstore): preserve dual inflight prefetch contract

Co-Authored-By: heihutu <[email protected]>

* test(ecstore): make prefetch assertion deterministic

Co-Authored-By: heihutu <[email protected]>

---------

Co-authored-by: heihutu <[email protected]>
2026-08-28 17:31:10 +00:00
Zhengchao AnandGitHub 75cd3885f3 fix(ecstore): reject Azure tier storageClass/spAuth instead of silently ignoring them (#6817)
TierAzure.storage_class and .sp_auth round-trip faithfully through the
admin API and on-disk config (ExternalTierAzure encode/decode in
tier.rs), so an operator can configure them, read them back via
ListTier, and never learn they do nothing. They are dropped only at the
WarmBackendAzure construction boundary: the Azure warm backend goes
through the same S3-compatible TransitionClient as every other
provider and has no Azure Blob-native client or Azure AD dependency
(confirmed: no azure_* crate anywhere in the workspace), so neither
field can actually be honored today. MinIO's reference implementation
(cmd/warm-backend-azure.go) treats both as first-class: storage_class
sets the blob access tier on every PUT, and sp_auth is a full
alternative to access/secret-key auth via azidentity, mutually
exclusive with it.

Rather than the larger, riskier options (add a native Azure SDK
dependency and a parallel non-S3 client path, or break the persisted
config format by removing the fields), this closes the silent-failure
gap with the minimal safe fix: TierConfigMgr::add now rejects an Azure
tier config with either field set, before backend construction,
returning ERR_TIER_INVALID_CONFIG with an explicit message instead of
accepting and ignoring. The fields stay in the config type (no format
break); already-persisted tiers with these fields set are grandfathered
in un-rejected (edit does not touch sp_auth or storage_class either).
Full support remains a larger follow-up if ever prioritized.

Also removes TierAzure::is_sp_enabled(), which had zero callers
repo-wide (backlog#2055 flagged this) and would have been misleading
dead weight once this decision was made — reusing it for the new gate
would also have been wrong, since it requires *all three* sp_auth
fields non-empty (&&), while the gate must reject on *any* one being set.

Refs rustfs/backlog#2055

(cherry picked from commit 8d148c4e9b2507a1c5075e3d9513adb8b5851ef5)
2026-08-28 17:12:30 +00:00
73c9dd4c9d fix(data-usage): preserve cold buckets in partial admin usage (#6811)
Merge newer partial observed usage into the complete authoritative admin baseline instead of replacing the full bucket set.

Keep the merged view partial and non-converged so shared consumers do not treat it as quota-authoritative.

Co-authored-by: heihutu <[email protected]>
2026-08-29 00:27:54 +08:00
Zhengchao AnandGitHub 2040f5aff9 fix(ci): pass --repo to gh release delete in preview cleanup (#6810) 2026-08-29 00:20:46 +08:00
唐小鸭andGitHub 5104be1d23 fix(ecstore): move conditional PUT lock to commit-time recheck (#6801)
A PUT with HTTP preconditions took the per-object namespace write lock
before ingesting the request body and held it until commit, so any
concurrent read of the same object queued behind client-paced body
ingestion until the 5s acquire timeout and surfaced as 503. Exposed as
a deterministic S3 Implemented Tests gate failure when #6770 routed
1 MB conditional writes onto the streaming path (rustfs/backlog#2074).

Keep a lock-free advisory precondition check before the body for fast
412/404, and evaluate the authoritative check under the put_object
commit lock, reusing the deferred shape data movement already uses.
Reads during ingestion now return the last committed version, and a
precondition invalidated mid-stream fails closed with 412 at commit.
2026-08-28 14:53:05 +00:00
5ef8b1ce5c fix(tier): harden reference proof and audit output (#6807)
Validate lifecycle tier references through the tier reference proof path, preserve S3 list CommonPrefix XML compatibility, and make GetObject audit completion use real S3 error status codes.

Co-authored-by: heihutu <[email protected]>
2026-08-28 22:46:00 +08:00
唐小鸭andGitHub ce4eca40a6 fix(site-replication): rotate-svc-acct no longer wedges replication (#6793) 2026-08-28 22:13:16 +08:00
Zhengchao AnandGitHub 86b6fecbb4 refactor(ecstore): migrate minio/r2/rustfs warm backends to shared S3 constructor (#6776) 2026-08-28 22:13:03 +08:00
hectorandGitHub 88b43f546f Extend functional test workflow with S3/KMS/tier suites (#6806) 2026-08-28 22:12:21 +08:00
Zhengchao AnandGitHub 2092fbf465 fix(s3-client): rename-align CommonPrefix for tier in-use XML parsing (#6805) 2026-08-28 14:11:04 +00:00
Zhengchao AnandGitHub ed66b0a04d refactor(ecstore): migrate huaweicloud/tencent warm backends to shared S3 constructor (#6775)
refactor(ecstore): huaweicloud/tencent reuse the shared S3 constructor

Migrates the Huaweicloud and Tencent tier warm backends onto the shared
S3-compatible constructor (backlog#2040). Also makes the shared
constructor's outbound-URL validation injectable per provider
(S3CompatibleWarmBackendParams::validate_endpoint) so it can centralize
rustfs/rustfs#6764's SSRF check for the providers that don't need an
exception, while accommodating rustfs/rustfs#6773's RustFS-specific
debug-only loopback opt-in without weakening the other six providers.

Updates scripts/error-other-format-baseline.txt: the one ::other(format!)
call site moves from the two per-provider files into the new shared
call site in warm_backend.rs (net call-site count unchanged).

Refs rustfs/backlog#2042
2026-08-28 13:12:55 +00:00
Zhengchao AnandGitHub 847fbd2a8b test(e2e): restore tier and inline full-suite checks (#6794) 2026-08-28 21:08:30 +08:00
7eddd1cf83 fix(heal): skip dangling delete grace failures (#6799)
Co-authored-by: heihutu <[email protected]>
2026-08-28 20:09:45 +08:00
唐小鸭andGitHub 2437069114 fix(iam): stop stamping the wall clock on policy-less group reads (#6791) 2026-08-28 19:50:24 +08:00
唐小鸭andGitHub 3b87d61cbf fix(site-replication): send the reverse-reachability probe as POST (#6790) 2026-08-28 19:50:09 +08:00
唐小鸭andGitHub eb6b617ca2 fix(sse): diagnose unresolvable encrypted metadata on reads (#6784) 2026-08-28 19:49:58 +08:00
Zhengchao AnandGitHub 64705d7589 refactor(ecstore): migrate aliyun/azure warm backends to shared S3 constructor (#6774) 2026-08-28 19:49:27 +08:00
Zhengchao AnandGitHub 206ef7d086 fix(s3): preserve atomic 1 MiB conditional writes (#6798) 2026-08-28 19:48:55 +08:00
b301834c6d chore(deps): update s3s revision (#6795)
Update the s3s git dependency to 6e7b41252c7ba218a90886f58d297716ddf68acf.

This pulls the upstream SelectRequest XML alias compatibility fix while keeping the RustFS s3s compatibility boundary intact.

Co-authored-by: heihutu <[email protected]>
2026-08-28 18:28:43 +08:00
028be4f604 refactor(heal): migrate mainline throttle to shared ForegroundPressure (#6780)
The heal manager carried its own byte-identical copy of the foreground pressure type and threshold computation that ecstore's data-movement backpressure also carries, so every change to the admission-utilization rules had to be mirrored by hand across two crates. The shared `ForegroundPressure` and `foreground_pressure` added to `rustfs-concurrency` now own that logic, and heal already depends on that crate, so this removes the duplicate without adding a crate edge.

`mainline_throttle_active` keeps the parts that are specific to this call site: the `mainline_throttle_enable` and both-thresholds-zero short circuit that avoids touching the provider at all, the optional-provider unwrap, and the heal-side threshold fields. Everything downstream is untouched — the `reason()` labels `foreground_read_pressure`, `foreground_write_pressure`, and `foreground_pressure` are byte-identical to the removed implementation, so the `rustfs_heal_mainline_throttle_total` reason label and the `heal_mainline_throttle` log fields keep their observability contract.

Refs rustfs/backlog#2049

(cherry picked from commit ec491bcbd8939e5978cd94f9a44cffb70d09fade)
(cherry picked from commit e800f29d6806591689204b3712300800b480beae)

Co-authored-by: houseme <[email protected]>
2026-08-28 08:21:27 +00:00
唐小鸭andGitHub 6f9adb3ad0 docs(kms): reconcile bulk-rekey contract with the shipped sweep (#6783) 2026-08-28 15:21:10 +08:00
Zhengchao AnandGitHub 19c7529d88 refactor(ecstore): migrate data movement backpressure to shared ForegroundPressure (#6779)
refactor(ecstore): use shared ForegroundPressure for data movement

The data movement backpressure module carried its own byte-identical copy of ForegroundPressure, its reason() label mapping, and the foreground utilization computation. rustfs-concurrency now owns that logic as workload::ForegroundPressure and workload::foreground_pressure, so the local copy was a cross-crate synchronization point that could silently drift from the heal-side and admission-side behavior.

Delete the local type and computation and call the shared function instead. The call site keeps what is specific to data movement: the config.enabled short circuit, the optional provider unwrap, and the read/write threshold percentages read from DataMovementBackpressureConfig. The reason() labels emitted into the rustfs_data_movement_backpressure_total metric and the data_movement_backpressure log event are unchanged, as are the existing tests and their assertions.

Refs rustfs/backlog#2048

(cherry picked from commit 6a26e144e06ced53a8dfd1712ab7aa24589646ff)
(cherry picked from commit ab5ab417e80179265c32b22a5e671eac0b9e43ae)
2026-08-28 15:11:08 +08:00
7951601ae8 perf(storage): optimize small-object GET/PUT paths (#6770)
* perf(ecstore): optimize small-object GET paths

Co-Authored-By: heihutu <[email protected]>

* perf(rustfs): optimize small-object request paths

Co-Authored-By: heihutu <[email protected]>

* chore(deps): upgrade argon2 and convert_case

Co-Authored-By: heihutu <[email protected]>

* fix(ecstore): restore reader hotpath attribution

Co-Authored-By: heihutu <[email protected]>

* test(ecstore): cover external mid-size fixtures

Co-Authored-By: heihutu <[email protected]>

---------

Co-authored-by: heihutu <[email protected]>
2026-08-28 15:10:33 +08:00
Zhengchao AnandGitHub b87ce6b183 fix(ci): sync Linux full E2E selection (#6787) 2026-08-28 15:01:30 +08:00
Zhengchao AnandGitHub f135583fee refactor(ecstore): migrate notify.rs default KVS tables to shared constructors (#6778)
refactor(ecstore): migrate notify default KVS to shared constructors

The amqp, nats, pulsar, redis, postgres, kafka and mysql default KVS tables in config/notify.rs duplicated the corresponding tables in config/audit.rs literally, leaving seven cross-file sync points that a future default or key-order edit had to keep aligned by hand. Replace those seven table bodies with calls to the shared constructors added in config/target_defaults.rs, passing the notify-side literals where the two subsystems genuinely differ: NOTIFY_REDIS_DEFAULT_CHANNEL for the redis channel and "rustfs_events" for the mysql table.

Key order is part of the admin config contract, so this is a pure restructuring: for all seven tables the ordered key sequence and every key's value and hidden_if_empty flag are unchanged.

DEFAULT_NOTIFY_WEBHOOK_KVS and DEFAULT_NOTIFY_MQTT_KVS are deliberately left untouched. Those two tables really do diverge between audit and notify, so folding them into shared constructors would change runtime behavior; the divergence is tracked separately in rustfs/backlog#2054.

Refs rustfs/backlog#2046

(cherry picked from commit 6df9b53027ef2f0cf9aa7b82ecb2af8c5108f11f)
(cherry picked from commit 14bbf756bea2ee6daacbfdc7d7a452effa649cff)
2026-08-28 15:01:19 +08:00
Zhengchao AnandGitHub 75e605fe87 refactor(ecstore): migrate audit.rs default KVS tables to shared constructors (#6777)
refactor(ecstore): migrate audit KVS defaults to shared constructors

The amqp, nats, pulsar, redis, postgres, kafka and mysql default KVS tables in config/audit.rs duplicated the corresponding tables in config/notify.rs, leaving seven cross-file sync points where a default could silently drift between the two subsystems. Build them from the shared constructors added in config/target_defaults.rs instead, passing in the two literals that are genuinely audit-specific: the redis pub/sub channel (AUDIT_REDIS_DEFAULT_CHANNEL) and the mysql destination table ("rustfs_audit_logs").

Key order, every default value and every hidden_if_empty flag are preserved exactly, since the key order drives the order admin config output lists keys in. DEFAULT_AUDIT_WEBHOOK_KVS and DEFAULT_AUDIT_MQTT_KVS are left untouched: those two tables really do differ from their notify counterparts, so unifying them would change runtime behavior.

Refs rustfs/backlog#2045

(cherry picked from commit 4de580e6d8901485eef268191924e035322d4d4e)
(cherry picked from commit e5e301fa78e7f27ffa85b05cf5e7962301a5ff00)
2026-08-28 15:01:09 +08:00
hectorandGitHub d115f1cbd7 test(pool): abort stale multipart uploads before decommission (#6771)
warp is killed at the write threshold and can leave in-flight multipart
uploads behind. rc.4-preview.1's decommission post-check refuses to
finalize a pool that still contains one (data is already moved, then the
pool is marked failed with 'resolve it before retrying'). Abort any
multipart uploads in the test bucket before starting decommission
(ListMultipartUploads + AbortMultipartUpload via the admin API).
2026-08-28 14:59:21 +08:00
cfaf87360f fix(storage): gate multipart upload part pressure (#6781)
Co-authored-by: heihutu <[email protected]>
2026-08-28 14:57:32 +08:00
Zhengchao AnandGitHub 876f60c1f4 fix(ci): restore tier e2e and locked builds (#6773)
* fix(tier): restore loopback e2e coverage safely

* fix(build): sync scanner dev dependency lock
2026-08-28 12:53:52 +08:00
Halil TeyfikandGitHub 488af5984c docs: fix Getting Started links (#6782) 2026-08-28 12:42:50 +08:00
GatewayJandGitHub 7136062c75 docs(agents): make branch naming identity-neutral (#6772) 2026-08-28 11:44:30 +08:00
Zhengchao AnandGitHub 1585308f0f fix(test): restore #[serial] markers the fallback runner still needs (#6767) 2026-08-28 08:41:36 +08:00
Zhengchao AnandGitHub e388a3ff53 fix(startup): never panic when the system CA bundle is absent (#6769) 2026-08-28 00:34:51 +00:00
Zhengchao AnandGitHub 22741603f5 test(e2e): finish the helper consolidation onto common.rs (#6766)
- common.rs gains an AdminTransport knob (Signed | Awscurl) with admin_execute_at plus three family wrappers: admin_create_user_via, admin_add_canned_policy_via, admin_attach_user_policy_via; the existing admin_create_user now delegates over the Signed transport.
- Deleted the four signed admin request clones in admin_mfa_test, admin_auth_test, reliant/tiering, and inline_fast_path_cluster_test; each keeps a thin local wrapper over common::admin_request so call sites keep their Option<&str> body shape.
- Deduped the notification_webhook signer onto common::signed_request and the webdav_core signer plus its three admin helpers onto the shared _via helpers.
- Consolidated the S3-client-with-credentials builders: admin_auth s3_client_with, existing_object_tag user_client/sts_session_client, bucket_policy_check create_user_client, and the create_user_s3_client copies in group_delete_test and replication_extension_test now delegate to create_s3_client_with_credentials / build_test_s3_config; replication_extension admin_add_canned_policy and admin_attach_policy_to_user route through the _via helpers on the Signed transport.
- The awscurl-gated suites (existing_object_tag_policy, bucket_policy_check, policy/policy_variables) keep going through the external awscurl binary via AdminTransport::Awscurl, preserving their wire behavior.

Part of rustfs/backlog#1846 (cluster 2).
2026-08-28 00:12:45 +00:00
Zhengchao AnandGitHub 3c89c71f66 fix(s3): round-trip null-version delete-marker identity (#6765)
* fix(s3): round-trip null-version delete-marker identity through listing and delete responses

On a versioning-suspended bucket, a null delete marker's identity was lost on the way back to the client at three points (issue #6745): ListObjectVersions advertised the marker's VersionId as the literal nil UUID instead of null; deleting by that id succeeded but the DeleteObjects/DeleteObject response reported the identity as null with no way to correlate it to the request; and the response lacked DeleteMarker/DeleteMarkerVersionId because the marker-ness comparison mixed the client-facing identity (Some(nil)) with the storage identity (None), so the removal also mis-recorded accounting and fired DeleteMarkerCreated semantics on later paths.

- Listing (bucket_usecase, s3_api/bucket, build_list_versions_next_marker) now maps the synthesized nil UUID to the literal null everywhere it reaches the wire, and VersionMarker::parse folds a nil-UUID marker from older listings into VersionMarker::Null so pagination resumes correctly.
- delete_objects normalizes both sides of the marker-ness comparison via delete_file_info_version_id (matching the adjacent explicit_delete_marker admission check) and reports DeleteMarkerVersionId as null for an explicit null-marker removal.
- resolve_delete_version_state reports delete_marker for an explicit-version delete whose target is a delete marker even when the bucket is versioning-suspended, fixing x-amz-delete-marker on the single-object path.
- The DeleteObjects response entry echoes the version identity the request addressed for marker removals, marker-removal accounting no longer records a marker creation, and notification events fire DeleteMarkerCreated only for actual marker creation.

Fixes #6745

* fix(s3): keep null-marker removal write shape undeleted and report marker semantics response-side

The first cut marked the storage delete request deleted for a null-marker removal, which FileMeta::delete_version interprets as the suspended-bucket delete-mints-a-marker write and re-creates the marker just removed. Carry marker-ness to responses via explicit_delete_removed_marker (single path) and a response-only branch flag (batch path) instead, keeping every storage write shape byte-identical to the pre-fix behavior. Adds an embedded end-to-end regression test covering the full issue #6745 round trip.
2026-08-27 23:55:05 +00:00
Zhengchao AnandGitHub db57fabcbd fix(tier): validate outbound URLs for all warm backend providers (#6764)
WarmBackendS3::new already rejects loopback, private, link-local, and
cloud metadata-service endpoints via validate_outbound_url, but the
Aliyun, Azure, Huaweicloud, Tencent, MinIO, R2, RustFS, and GCS warm
backend constructors built their transition clients directly from
conf.endpoint without the same check.

The endpoint comes from the AddTier admin API, gated only by
SetTierAction, which can be a narrower IAM grant than root. Any
principal holding it could point one of these eight tier types at an
internal address (loopback, RFC1918, link-local, or a cloud metadata
IP) and have the server issue authenticated outbound requests to it, a
server-side SSRF vector that the S3 and Wasabi tier types were already
closed against.

Apply the same validate_outbound_url check at construction time for
all eight providers, before any credentials or network client are
built, mirroring the existing WarmBackendS3 pattern. GCS keeps its
default-endpoint behavior when conf.endpoint is empty and only
validates an explicitly configured endpoint.

Add a regression test per provider asserting that a loopback endpoint
is rejected before any backend/network setup, matching the existing
WarmBackendS3 coverage.

Update the error(format!) ratchet baseline: these are one-shot admin
tier-configuration validation errors returned once per AddTier call,
not per-disk I/O errors that flow through reduce_errs quorum
aggregation (backlog#1845), so the new ::other(format!) call sites do
not introduce a quorum-bucketing hazard. They mirror the pre-existing,
already-baselined warm_backend_s3.rs call site.
2026-08-27 23:37:50 +00:00
Zhengchao AnandGitHub 03888bd266 chore(tier): remove dead trailing_headers config from warm backends (#6762)
TransitionClient::new() in crates/s3-client/src/transition_api.rs computes
trailing_header_support = opts.trailing_headers && override_signer_type == SignatureV4,
but override_signer_type is hardcoded to SignatureDefault at construction
and never mutated afterwards, so the expression is always false regardless
of opts.trailing_headers. The resulting field also has no live reader: its
only reference is inside PutObjectOptions::validate() in
crates/s3-client/src/api_put_object.rs, which is itself
#[allow(dead_code, reason = "MinIO-parity ... no caller in this port")],
and even there the reference to trailing_header_support is commented out.

So trailing_headers: true in the seven warm_backend_*.rs constructors has
never had any effect on request signing or chunked/trailing-header
behavior (stream_sha256 signing is gated separately by
metadata.stream_sha256 && !self.secure). Remove the misleading dead
configuration from the seven provider constructors so it doesn't look
like intentional, load-bearing behavior to future readers.

Found during adversarial self-check while implementing rustfs/backlog#2040 (out of that issue's scope).
2026-08-28 07:35:48 +08:00
Zhengchao AnandGitHub a6a04b5faa refactor(ecstore,rustfs): reuse canonical starts_with_ignore_ascii_case (#6759)
* refactor(ecstore,rustfs): reuse canonical starts_with_ignore_ascii_case

`crates/utils/src/http/metadata_compat.rs` owns the internal metadata key helpers, including `starts_with_ignore_ascii_case`. Two files carried their own byte-identical copies of that predicate: `SetDisks::starts_with_ignore_ascii_case` in ecstore and a free function in the S3 options layer. Both drive internal metadata key classification (`internal_metadata_suffix` and quorum hashing on one side, `should_skip_object_metadata_key` and `is_reserved_user_metadata_key` on the other), so keeping three implementations of one predicate is an avoidable drift risk on a path that decides whether an internal key is treated as user metadata.

Delete both local copies and call the canonical implementation. Every prefix used at these call sites is an ASCII constant or literal, where the canonical byte-slice comparison and the removed `str::get(..n)` form are equivalent; that equivalence was checked differentially over 4.6M (key, prefix) pairs, including keys with multi-byte characters straddling the prefix boundary. No other logic in `internal_metadata_suffix` or `should_skip_object_metadata_key` changed.

Add regression tests on both sides pinning the two properties the switch depends on: internal prefixes match case-insensitively (a mixed-case `X-RustFS-Internal-*` key stays internal), and keys shorter than a prefix never match (they stay ordinary user metadata).

Refs rustfs/backlog#2051

* fix(rustfs): avoid typos-checker false positive in prefix-length test

The test literal "x-rustfs-encryptio" (a deliberate truncation of the
x-rustfs-encryption- prefix, used to assert that a key shorter than every
internal prefix falls through to user metadata) reads as a likely typo of
"encryption" to the repo's typos CI check. Derive it from
RUSTFS_ENCRYPTION_PREFIX via slicing instead of a hand-typed literal, which
both satisfies the linter and ties the truncation to the real constant
instead of a copy-typed guess.

Refs rustfs/backlog#2051
2026-08-28 07:35:09 +08:00
Zhengchao AnandGitHub 18068eb7e5 fix(deps): move off the yanked chacha20 0.10.1 (#6768)
chacha20 0.10.1 was yanked on crates.io today, which fails the Cargo Deny gate (error[yanked]) on every branch. cargo update -p chacha20 to 0.10.2; no API change, all dependents are semver-compatible.

Verification: cargo check -p rustfs-crypto; the Cargo Deny job on this PR is the authoritative gate.
2026-08-28 07:03:24 +08:00
Zhengchao AnandGitHub 921a48bd14 refactor(lifecycle): reuse the replication tag parser (#6761)
`crates/lifecycle/src/tagging.rs` carried a byte-identical copy of the `form_urlencoded` tag decoder already owned by `rustfs-replication`, plus a duplicate of its test. Since `crates/lifecycle` already depends on `rustfs-replication`, replace the copy with a `pub(crate) use` re-export: no new crate edge, one parser, and no second implementation to drift from the replication contract. The `rule.rs` call site is unchanged.

Also drop `crates/ecstore/src/bucket/lifecycle/tagging_boundary.rs`, a migration-era boundary shim with zero call sites in the tree.
2026-08-27 22:56:13 +00:00
Zhengchao AnandGitHub 2e6511566e refactor(rustfs): consolidate bucket metadata import match arms (#6760)
The import_bucket_metadata handler carried eight match arms whose bodies were
byte-identical apart from the type a payload is validated against and the pair
of BucketMetadata fields it lands in, so every arm repeated the same warn! call
and the same metadata lookup. Fold them into one or-pattern arm backed by
apply_imported_bucket_config, where a single conf_name match owns both the
validated type and the destination field pair and can no longer drift apart.

Validation still runs before the metadata lookup, the warn! event, fields, and
label are unchanged, and the BUCKET_POLICY_CONFIG and BUCKET_QUOTA_CONFIG_FILE
arms keep their own handling. Regression tests drive the full mapping table:
each config file's payload lands only in the field it owns, an unparsable
payload leaves the field untouched, and a rejected entry does not stop the
remaining ones from being imported.

Refs rustfs/backlog#2052
2026-08-27 22:55:09 +00:00
Zhengchao AnandGitHub ec8abb19ab refactor(s3-client): reuse rustfs-utils header classification instead of duplicating it (#6758)
crates/s3-client/src/utils.rs carried a verbatim copy of the header
classification tables and predicates owned by
crates/utils/src/http/headers.rs: SUPPORTED_HEADERS (same 11 keys),
SUPPORTED_QUERY_VALUES (same 9 keys), and is_standard_header /
is_storageclass_header / is_amz_header / is_rustfs_header /
is_minio_header with byte-identical bodies. The duplication was already
half-resolved and inconsistent — the local is_amz_header called
rustfs_utils::http::is_sse_header while consulting its own tables — and
s3-client already depends on rustfs-utils with the "full" feature, so
reusing the canonical owner adds no crate edge.

The sole caller, PutObjectOptions::header(), now imports the five
predicates from rustfs_utils::http. Semantics are unchanged: both sides
normalize with to_lowercase(), return false for unknown keys, and the
storage-class constants are the same string ("x-amz-storage-class" from
s3s::header::X_AMZ_STORAGE_CLASS vs rustfs_utils AMZ_STORAGE_CLASS), so
the set of user-metadata headers passed through verbatim rather than
prefixed with x-amz-meta- is identical.

SUPPORTED_QUERY_VALUES is deleted outright: s3-client had no reader for
it (utils consumes its own copy via is_standard_query_value). The
base64_encode/base64_decode helpers and their rustfs/rustfs#4811
regression test stay untouched, and lazy_static remains a dependency
because crates/s3-client/src/constants.rs still uses it.

Refs rustfs/backlog#2050
2026-08-28 06:53:15 +08:00
Zhengchao AnandGitHub 2a8be5566d refactor(concurrency): consolidate ForegroundPressure into workload owner (#6757)
ForegroundPressure had two definitions with byte-identical pressure computation: one in ecstore data-movement backpressure and one in the heal manager queue. That duplication is a violation of the ARCHITECTURE.md invariant that each type has exactly one definition, and it means any future change to the utilization math has to land twice.

Add the canonical `ForegroundPressure` and a `foreground_pressure(snapshot, read_threshold_pct, write_threshold_pct)` function to `crates/concurrency/src/workload.rs`, which already owns `WorkloadClass`, `AdmissionState`, and the admission snapshot contract. Both existing consumers already depend on `rustfs-concurrency`, so no crate edge is added.

The filter_map pipeline is transferred verbatim, preserving all five boundary behaviors (zero threshold, zero limit, missing entry, missing active count, and the `Saturated` full-utilization special case), the mul-before-div percentage normalization, the `>=` threshold comparison, and the read-then-write ordering that makes `max_by_key` break utilization ties toward the write class. The enable switch is deliberately left out: ecstore gates on `config.enabled` while heal gates on `mainline_throttle_enable` plus a both-thresholds-zero check, so each call site keeps its own condition.

This is the expand step only. The ecstore and heal copies are untouched and are removed by the follow-up migrate task.

Refs rustfs/backlog#2047
2026-08-28 06:52:50 +08:00
Zhengchao AnandGitHub bcbc58b6a0 refactor(ecstore): extract shared warm backend S3 constructor (#6755)
The seven S3-compatible warm backend providers (Aliyun, Azure, Huaweicloud, Tencent, MinIO, R2, RustFS) each carry a byte-identical copy of the same statically-credentialed TransitionClient construction and of the same optimal_part_size helper. Add both to the module that already owns the WarmBackend trait and WarmBackendS3, so the per-provider migrate step can drop its duplicate without redesigning anything.

bucket_lookup is a parameter rather than a constant because the providers split into two families: Aliyun, Azure, Huaweicloud, and Tencent pin BucketLookupDNS, while MinIO, R2, and RustFS leave it at the BucketLookupAuto default. Hardcoding either value would silently change bucket addressing for the other family during the migrate step.

Error texts, validation order, prefix and host/port normalization are reproduced exactly from the Aliyun/MinIO family. No provider file is touched and no production caller exists yet, so the new unit tests are the first callers.

Refs rustfs/backlog#2040
2026-08-27 22:16:47 +00:00
Zhengchao AnandGitHub 28fa412a06 refactor(ecstore): extract shared audit/notify KVS table constructors (#6756)
The audit and notify subsystems each declare their own default KVS table for the same nine delivery targets. For amqp, nats, pulsar, postgres and kafka the two declarations are byte-identical; for redis and mysql they differ only in a single default literal (the pub/sub channel and the destination table). Keeping two copies means every default or key-order change has to be made twice, and a missed edit silently changes what admin config reports for one subsystem only.

Add `config::target_defaults` with one constructor per shared table, taking the diverging literal as a parameter for redis and mysql, plus a small `kv` helper that replaces the repeated `KV { .. }` literals. Key order is reproduced exactly because it drives the order the admin API lists keys in. Unit tests pin the full ordered key/value/hidden_if_empty triple of every table against hard-coded literals, and cover both the audit and the notify literal for the two parameterized tables.

Webhook and mqtt are deliberately left out: audit's webhook table carries extra batching and retry keys, both webhook tables disagree on key order and on the auth-token hidden_if_empty flag, and mqtt disagrees on qos, keep-alive interval and reconnect interval. Those are real behavioral forks, not duplication, so they stay declared in place.

This is the expand step only. Nothing calls the new module yet, so audit.rs and notify.rs are untouched and no default changes; the constructors carry an item-level allow(dead_code) until the migrate step points both files at them.

Refs rustfs/backlog#2044
2026-08-27 22:16:03 +00:00
6f7a4ff060 fix(api): preserve server-side storage error surface (#6753)
Co-authored-by: heihutu <[email protected]>
2026-08-27 15:42:51 +00:00
hectorandGitHub 8a57632bfd ci: use dedicated RUSTFS_PERF_NODES for performance test (#6754) 2026-08-27 22:55:26 +08:00
GatewayJandGitHub 2eb4ddf4af test(table-catalog): automate DuckDB REST conformance (#6750)
* test(table-catalog): automate DuckDB REST conformance

* fix(table-catalog): protect DuckDB smoke tables
2026-08-27 22:34:10 +08:00
GatewayJandGitHub e281ed2f6d test(table-catalog): generate DuckDB REST attach SQL (#6749) 2026-08-27 22:25:33 +08:00
hectorandGitHub 2e6c820f53 test(heal): relative disk target and fail fast on terminal-but-short (#6748)
* test(heal): relative disk target and fail fast on terminal-but-short

The absolute 40 GiB heal target was calibrated to the background scanner
(auto-heal), which is now disabled for determinism; with only the explicit
heal the recovered node lands at ~36 GiB for 40 GiB survivors. Make the
success criterion relative: the outage node must reach at least 90% of the
least-used surviving node (absolute HEAL_TARGET_GB floor optional, default
0 = relative only).

Also fail fast when the heal task reaches a terminal success but the disk
target is not met (previously the monitor kept polling until timeout), and
drop the misleading 'progress absent' warning on the final (cleaned) task
response — mid-run progress is reported correctly.

Validated live: heal summary=finished, 0 failed, vm000/vm001=40GB,
vm002=40GB (target 36GB), test PASSED.

* test(heal): gate success on server verdict + data read-back, drop disk GB gate

The per-node disk-usage target (40 GiB / 90% of survivors) is not a
code-level invariant: EC distributes different shards per node, so the
final GB per node depends on the layout, not on heal correctness. Gate the
test on what the server actually verifies:

- Heal task terminal success (finished/completed) with objectsFailed == 0
  (the server's per-object scan/repair verdict).
- S3 read-back verification: list the test bucket and GET a sample of
  objects, requiring HTTP 200 for every read (end-to-end proof the data is
  still reconstructable after repair). The GET uses a discard mode so
  binary bodies are not captured (no null-byte warnings / SIGPIPE).

Per-node disk usage stays in the output as observability (with a warning if
the outage node gained no usage), not as the pass/fail gate. Removes the
heal_target_gb input and the relative-target logic.

Validated live: heal summary=finished, 0 failed, 20/20 objects read back,
vm002_used=40GB, PASS.
2026-08-27 22:25:17 +08:00
hectorandGitHub d48dda5bdc ci: add RustFS 4x4 performance test workflow and scripts (#6752)
* ci: add RustFS 4x4 performance test workflow and scripts

* ci: run performance test on dedicated pf-testing runner
2026-08-27 22:24:57 +08:00
44f3f0e73e perf(storage): gate large foreground PUT pressure (#6751)
Add a default-on, size-aware foreground PUT admission policy so large or
unknown-size PutObject requests are backpressured before body ingest and
erasure/RPC fan-out. Preserve the explicit strict gate semantics, including
limit=0 as an opt-out, and keep small PUTs on the legacy fast path.

Closes rustfs/backlog#2038

Co-authored-by: heihutu <[email protected]>
2026-08-27 22:06:51 +08:00
152f110583 revert: rollback AHashMap changes and keep using std HashMap (#6741)
* Revert "perf(ecstore): use AHashMap for FileInfo metadata fields (#6738)"

This reverts commit 13a2ae212e.

* fix(filemeta): restore standard HashMap metadata (#6742)

Remove the direct ahash dependency added for FileInfo metadata and revert the affected filemeta/ecstore call sites back to std::collections::HashMap.

Co-authored-by: heihutu <[email protected]>

---------

Co-authored-by: heihutu <[email protected]>
2026-08-27 20:46:08 +08:00
hectorandGitHub f1de19fc14 test: fall back to writable temp files for logs/status in /tmp (#6743)
A fixed /tmp path (log file, final heal status, warp log) can be owned by
another user on the shared runner (e.g. a previous root run), which made the
github-runner user fail: tee could not append the test log, the final heal
status write killed step 6 with EACCES, and upload-artifact could not read
stale root-owned warp logs. The heal scenario itself had passed
(summary=finished, vm002 reached the target) before the status-save died.

- Log files fall back to a unique mktemp path when the configured path is not
  writable (heal + pool scripts).
- The final heal status is written to a mktemp file (best effort).
- Workflow artifact uploads use globs for the fallback names.
2026-08-27 19:16:52 +08:00
唐小鸭andGitHub 7c4e514ec9 fix(sse): document and lock anonymous denial under KMS key policy (#6739) 2026-08-27 18:34:34 +08:00
housemeandGitHub 13a2ae212e perf(ecstore): use AHashMap for FileInfo metadata fields (#6738) 2026-08-27 18:34:10 +08:00
cxymdsandGitHub 94a6da6e83 feat(s3): enforce multipart presigned size limits (#6732) 2026-08-27 18:33:56 +08:00
hectorandGitHub a199312e45 test(heal): add node-outage heal E2E script and workflow (#6733)
* test(heal): add node-outage heal E2E script and workflow

RustFS heal test on the 3x4 cluster (3 nodes x 4 disks, same
RUSTFS_VOLUMES expression on every node): write data with warp, stop the
outage node mid-write, restart it, start cluster heal via the admin API,
and pass only when the heal task finishes with 0 failures AND the outage
node's disk usage reaches the target.

Includes the GitHub Actions workflow (smoke-testing runner, nightly deb by
default) and a README. Validated end-to-end on the test environment:
40/40/16 GiB before heal -> 40/40/40 GiB after heal, summary=finished.

The script also writes RUSTFS_HEAL_TASK_TIMEOUT_SECS (default 6h) into the
node config because the server default (5 min) is far too short for
healing tens of GiB.

* ci(pool-test): chain heal regression after the pool test

The pool-expansion workflow is now triggered by the Nightly GNU Build
(workflow_run, replacing the schedule) and runs two sequential jobs on the
shared test environment:

1. pool-expansion-test (existing) — skipped if the nightly build failed.
2. heal-test — runs after the pool test regardless of its outcome
   (if: always()): a pool failure makes the run red but does not block the
   heal regression. Runs the heal script (reset -> install/start 3x4 ->
   write/outage -> heal -> verify -> reset).

* test(heal): address review — camelCase progress, fail-closed, workflow hygiene

- Heal progress fields are camelCase in the API (objectsScanned/objectsHealed/
  objectsFailed/progressPercentage); read them with a snake_case fallback and
  distinguish null (absent) progress from zero, logging null as evidence
  (rustfs/backlog#2035) instead of silently coercing.
- Fail closed in step 3: the outage node must actually be inactive after stop,
  the write target must be reached, and an unobserved outage or incomplete
  write fails the test instead of warning.
- Step 4 waits (bounded) for the cluster to report an active pool after the
  outage-node restart instead of swallowing the verification error.
- Heal start fails fast on 400/403 (deterministic request/auth problems) and
  only retries transient server errors.
- Disable the background scanner (RUSTFS_HEAL_AUTO_HEAL_ENABLE=false) so the
  explicit heal is the only repair mechanism and the outage is observable.
- Workflows: heal and pool share one concurrency group; workflow_run requires
  an exact successful nightly conclusion; checkout is pinned to the triggering
  SHA; comma-separated step args are quoted (actionlint SC2054).
2026-08-27 18:27:13 +08:00
3420006762 fix(health): bound remote lock online checks (#6737)
Keep /health/ready from riding the generic internode lock RPC and channel keepalive budgets when a peer host is unreachable. Add a health-specific lock online timeout, route ping failures through the existing remote lock RPC eviction path, cache the static ping payload for readiness fan-out, and cover hanging cached channels with focused tests.

Refs rustfs/backlog#2033

Refs rustfs/rustfs#6286

Co-authored-by: heihutu <[email protected]>
2026-08-27 18:09:37 +08:00
唐小鸭andGitHub c95b4f0820 test(ecstore): make config tests robust to DEFAULT_KVS registration order (#6731)
Config::new() and the external decode path read the process-global
DEFAULT_KVS OnceLock at call time, and config::tests in the same test
binary register it via crate::config::init() mid-run. Several com.rs
tests asserted on unregistered state (heal section absence, equality
with a later Config::new()), so they could flip depending on thread
scheduling under cargo test -p rustfs-ecstore --lib config::.

Assert on the semantic heal diff instead of section presence, normalize
compared configs with a single DEFAULT_KVS snapshot taken after both
sides exist, and compare the snapshot transaction test against the
persisted baseline bytes.
2026-08-27 16:36:13 +08:00
d9080ae77f test(pool): cover rebalance retry and cold-start recovery (#6720)
* test(pool): fix warp log path and retry rebalance start

- warp writes now use a unique mktemp log file instead of a fixed
  /tmp/rustfs-warp.log: the runner user could not write the stale
  root-owned file, which made the background warp process die instantly
  (warp never ran). The workflow uploads /tmp/rustfs-warp.*.log.
- rebalance start is retried (6x, 20s apart): nightly builds gate
  rebalance activation on a live cross-pool fence fleet capability proof
  that takes ~10-20s to re-establish after a pool joins. Verified live:
  attempt 1 fails with 500 'pool activation requires a live fleet
  capability proof', attempt 2 succeeds.

* test(pool): annotate known server-side issues in failure output

When a node fails to start, grab the rustfs journal tail and match known
server-side error signatures (e.g. the fleet capability proof cold-start
regression, rustfs/backlog#2031), printing a hint with the tracking issue.
Also annotate the rebalance-start retry exhaustion and the rc.3 decommission
metacache-listing failure with actionable guidance.

* fix(ecstore): defer rebalance activation without fleet proof

---------

Co-authored-by: 马登山 <[email protected]>
Co-authored-by: cxymds <[email protected]>
2026-08-27 16:18:39 +08:00
Zhengchao AnandGitHub 95c926dc79 ci(release): delete preview releases after the deliverable publishes (#6729)
Preview tags stay as the traceability record for the validated commit, but their GitHub Releases are internal validation state and should not accumulate on the Releases page next to real deliverables.

Add a cleanup-preview-releases job that runs after publish-release succeeds for a release or prerelease tag and deletes every Release whose tag is exactly <target>-preview.<digits>. The tags themselves are kept: the job never passes --cleanup-tag. Tag matching uses jq string operations rather than a regex over the version, so dots in the version cannot widen the match, and the release listing is fetched before filtering so an API failure aborts the job instead of looking like there was nothing to clean up.

Extend the preview release workflow guard with the job condition, the delete invocation, both tag-matching filters, and an absent check for --cleanup-tag.
2026-08-27 16:18:05 +08:00
156 changed files with 10788 additions and 4531 deletions
@@ -6,7 +6,7 @@ description: "Run the end-to-end RustFS console gate, version bump, preview vali
This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. That Release is temporary: `build.yml` deletes it automatically once the final tag's Release is published, so the Releases page ends up carrying deliverables only while the `-preview.N` tags stay behind as the traceability record. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
Pipeline shape:
@@ -19,6 +19,7 @@ check console main against its latest Release
-> validate with latest rc client
-> report preview acceptance results -> STOP for explicit human confirmation
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
-> CI deletes the <target>-preview.N Releases (tags kept)
```
On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
@@ -51,14 +52,16 @@ Rules:
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
## Hard rules
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
- Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
- Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
- Completing preview acceptance does not authorize the final tag. After Phases 35 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
@@ -230,6 +233,7 @@ git push origin "<target>"
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
## Output contract
@@ -239,5 +243,5 @@ Always report:
- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
- Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
- Any deviation from this pipeline and why the user approved it.
+1 -1
View File
@@ -1,2 +1,2 @@
sha256-darwin=d6aa36cfaae2c4d8590482c7e47138c5965b335b34a75f50d11ffc3366e9021e
sha256-linux=c8315465f50c194faee36141cdbb1e15e59271e524d948564a69e2d5eb408f2a
sha256-linux=96db8060fce98addda4f69092d297ca236bec4892d820617a26a261eedac61b0
+1 -1
View File
@@ -1 +1 @@
sha256=655a3f3c1d042e694339d15caba7580518320322d1bac0f09450b37e6c09e2e7
sha256=8d5517f5f2fc32d561782dfccd51b7f746f5e25b2835e37e100c883f7f18777d
+1
View File
@@ -5,3 +5,4 @@ self-hosted-runner:
- sm-standard-2
- sm-standard-4
- dind-sm-standard-2
- smoke-testing
+5
View File
@@ -7,6 +7,11 @@
{ "workflow": ".github/workflows/e2e-s3tests.yml", "max_age_hours": 192 },
{ "workflow": ".github/workflows/fuzz.yml", "max_age_hours": 36 },
{ "workflow": ".github/workflows/mint.yml", "max_age_hours": 192 },
{
"workflow": ".github/workflows/minio-interop.yml",
"max_age_hours": 36,
"never_ran_grace_until": "2026-09-08T00:00:00Z"
},
{ "workflow": ".github/workflows/nightly-gnu.yml", "max_age_hours": 36 },
{ "workflow": ".github/workflows/performance-ab.yml", "max_age_hours": 36 },
{
+49
View File
@@ -1033,6 +1033,55 @@ jobs:
echo "🎉 Released $TAG successfully!"
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
# Remove the internal preview releases once the deliverable release is live.
# Only the Releases are deleted; the -preview.N tags stay so the validated
# commit remains traceable.
cleanup-preview-releases:
name: Cleanup Preview Releases
needs: [ build-check, publish-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: write
steps:
- name: Delete preview releases for this target
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
set -euo pipefail
TAG="${{ needs.build-check.outputs.version }}"
RELEASES_JSON="${RUNNER_TEMP}/releases.json"
# Fetch before filtering: a failed listing must abort here instead of
# looking like "nothing to clean up".
gh api --paginate "repos/${GITHUB_REPOSITORY}/releases?per_page=100" > "$RELEASES_JSON"
# Match only <target>-preview.<digits>. String operations, not a
# regex over the tag, so dots in the version cannot widen the match.
DELETED=0
while IFS= read -r preview_tag; do
[[ -n "$preview_tag" ]] || continue
echo "🧹 Deleting preview release $preview_tag (tag kept)"
gh release delete "$preview_tag" --repo "${GITHUB_REPOSITORY}" --yes
DELETED=$((DELETED + 1))
done < <(
jq -r --arg tag "$TAG" '
.[]
| select(.tag_name | startswith($tag + "-preview."))
| select(.tag_name | ltrimstr($tag + "-preview.") | test("^[0-9]+$"))
| .tag_name
' "$RELEASES_JSON"
)
if [[ "$DELETED" -eq 0 ]]; then
echo "️ No preview releases to clean up for $TAG"
else
echo "✅ Removed $DELETED preview release(s) for $TAG"
fi
alert-on-failure:
name: Alert on scheduled failure
needs: [build-check, prepare-platform-matrix, build-rustfs, build-summary]
+16 -16
View File
@@ -20,27 +20,27 @@
# each run with Docker and then runs the `#[ignore]` reader tests in
# rustfs/src/storage/minio_generated_read_test.rs.
#
# Scope: end-to-end MinIO-to-RustFS SSE interop is NOT implemented yet. Both
# envelope parsers reject MinIO's own wrapped-DEK shape — see
# `is_data_key_envelope` in crates/kms/src/encryption/dek.rs and the
# `deny_unknown_fields` `LocalSseDekEnvelope` in rustfs/src/storage/sse.rs — and
# closing that gap is tracked in rustfs/backlog#1638. Treat this job as the
# harness for #1638, not as standing evidence that a MinIO migration reads back.
# Scope: MinIO-to-RustFS SSE read interop is implemented behind the `rio-v2`
# feature for MinIO's builtin static-KMS deployments — SSE-S3 and SSE-KMS
# (single- and multipart) since rustfs/rustfs#6191, SSE-C detection since the
# rustfs/backlog#1638 D2 close-out. This job is the standing evidence: it
# regenerates real MinIO backend trees and proves byte-identical plaintext
# reconstruction. KES/MinKMS-backed MinIO objects remain unreadable by design
# (their envelopes are sealed by the KES service, not by a key RustFS can
# hold), and default RustFS builds do not include the read path — it is a
# special-purpose migration capability, not a default-build feature.
#
# Runner: GitHub-hosted `ubuntu-latest`. It reliably ships Docker + Python,
# unlike the self-hosted fleet, whose pods drift in Docker/pip availability
# (see the infra note in e2e-s3tests.yml). Nightly + manual only.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
# While disabled, this workflow is deliberately absent from
# .github/scheduled-validations.json — a disabled workflow can never satisfy the
# freshness check. Whoever re-enables it must re-add the entry in the same
# change so the freshness gate covers it again.
# Enablement: this workflow was long disabled in the repository's Actions
# settings (state: disabled_manually — a state that lives in GitHub's UI and is
# invisible in this file). The change that updated this banner also re-added
# the .github/scheduled-validations.json entry; both only make sense together
# with re-enabling the workflow in the Actions settings. If it is ever disabled
# again, remove the scheduled-validations entry in the same change — a disabled
# workflow can never satisfy the freshness check. See rustfs/backlog#1603.
#
name: minio-interop
+134
View File
@@ -0,0 +1,134 @@
name: RustFS Heal Test
on:
workflow_dispatch:
inputs:
package_url:
description: 'Direct .deb URL (nightly/R2). Defaults to the latest nightly deb.'
required: false
type: string
stop_node_gb:
description: 'Stop the outage node when surviving nodes reach N GiB'
required: false
default: '15'
warp_stop_gb:
description: 'Stop warp when surviving nodes reach N GiB'
required: false
default: '40'
cleanup_before:
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
type: boolean
default: true
cleanup_after:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
workflow_run:
# Run after the nightly build completes: the nightly deb is what the test
# installs. Heal also runs inside the functional suite; this standalone
# workflow enables manual single-suite runs as well.
workflows: ["Nightly GNU Build"]
types: [completed]
permissions:
contents: read
# Only one test at a time: both this and the pool-expansion workflow mutate
# the same test environment, so they share one concurrency group.
concurrency:
group: rustfs-pool-expansion-test
cancel-in-progress: false
defaults:
run:
shell: bash
env:
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
RUSTFS_API_ENDPOINT: ${{ secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }}
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
jobs:
heal-test:
runs-on: smoke-testing
timeout-minutes: 480
# Run on manual dispatch, or when the nightly build completed successfully.
# Skipped when nightly failed.
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
steps:
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/auto-testing
ref: main
path: auto-testing
persist-credentials: false
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Show environment
run: |
uname -a
jq --version
openssl version
warp --version || true
df -h /data | tail -1
- name: Reset test environment (before)
if: ${{ inputs.cleanup_before != 'false' }}
run: |
chmod +x auto-testing/rustfs_heal_test.sh
./auto-testing/rustfs_heal_test.sh --reset -y
- name: Install RustFS package & start cluster
run: |
ARGS=(--steps "1,2" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}"
- name: Preflight checks
run: |
ARGS=(--preflight --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}"
- name: Run heal test (write -> outage -> heal -> verify)
run: |
./auto-testing/rustfs_heal_test.sh \
--steps "3,4,5,6,7" -y \
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
--stop-node-gb "${{ inputs.stop_node_gb }}" \
--warp-stop-gb "${{ inputs.warp_stop_gb }}" \
--log-file /tmp/rustfs-heal-test.log
- name: Upload test logs
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: rustfs-heal-test-${{ github.run_id }}
path: |
/tmp/rustfs-heal-test*.log
/tmp/rustfs-warp.*.log
if-no-files-found: warn
- name: Reset test environment (after)
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
./auto-testing/rustfs_heal_test.sh --reset -y
- name: Notify on failure
if: failure()
run: |
echo "RustFS heal test failed"
echo "Package source: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
echo "See the uploaded log artifact for details."
@@ -0,0 +1,206 @@
name: RustFS Performance Test
on:
workflow_dispatch:
inputs:
package_url:
description: 'Direct .deb URL (nightly/R2). Defaults to the latest nightly deb.'
required: false
type: string
test_method:
description: 'Benchmark method(s) to run (manual runs only; "all" = GET+PUT+MIXED)'
type: choice
options:
- all
- get
- put
- mixed
default: 'all'
object_size:
description: 'Object size(s) to test (manual runs only; "all" = all 10 sizes)'
type: choice
options:
- all
- 1KiB
- 4KiB
- 16KiB
- 128KiB
- 1MiB
- 4MiB
- 8MiB
- 16MiB
- 32MiB
- 64MiB
default: 'all'
warp_duration:
description: 'warp duration per round (e.g. 5m, 30s)'
required: false
default: '5m'
warp_concurrency:
description: 'warp concurrency'
required: false
default: '64'
cleanup_before:
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
type: boolean
default: true
cleanup_after:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
workflow_run:
# Run after the nightly build completes; the nightly deb is what the test installs.
workflows: ["Nightly GNU Build"]
types: [completed]
permissions:
contents: read
# Dedicated pf-testing runner/environment: own concurrency group so perf runs
# never block (or are blocked by) the pool-expansion / heal tests.
concurrency:
group: rustfs-performance-test
cancel-in-progress: false
defaults:
run:
shell: bash
env:
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
# Performance test uses its own node list (4 nodes); the shared
# RUSTFS_NODES secret is used by the 3-node pool-expansion / heal tests.
RUSTFS_NODES: ${{ secrets.RUSTFS_PERF_NODES || vars.RUSTFS_PERF_NODES || 'vm000 vm001 vm002 vm003' }}
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
# Package used by the nightly run (workflow_dispatch inputs are empty for
# workflow_run events), i.e. the latest nightly deb published by nightly-gnu.yml.
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
# Fixed benchmark result directory so later steps can read summary.md
RUSTFS_RESULT_DIR: /tmp/rustfs-perf-results
# Cross-repo token for writing to rustfs/backlog (set in repo settings)
PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
jobs:
performance-test:
runs-on: pf-testing
timeout-minutes: 900
# Run on manual dispatch, or when the nightly build completed successfully.
# Skipped when nightly failed.
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
steps:
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/auto-testing
ref: main
path: auto-testing
persist-credentials: false
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Show environment
run: |
uname -a
jq --version
warp --version || true
df -h /data | tail -1
- name: Reset test environment (before)
if: ${{ inputs.cleanup_before != 'false' }}
run: |
chmod +x auto-testing/rustfs_performance_test.sh
./auto-testing/rustfs_performance_test.sh --step 1 -y
- name: Install RustFS package & start cluster (4x4)
run: |
ARGS=(--steps "2,3,4" -y)
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./auto-testing/rustfs_performance_test.sh "${ARGS[@]}"
- name: Preflight checks
run: |
ARGS=(--preflight)
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./auto-testing/rustfs_performance_test.sh "${ARGS[@]}"
- name: Run benchmark (GET/PUT/MIXED)
id: benchmark
run: |
# Empty on automatic (workflow_run) runs -> full 30 rounds.
# Manual dispatch can restrict method(s)/size(s).
export WARP_METHODS="${{ inputs.test_method }}"
export WARP_SIZES="${{ inputs.object_size }}"
./auto-testing/rustfs_performance_test.sh \
--step 5 -y \
--warp-duration "${{ inputs.warp_duration || '5m' }}" \
--warp-concurrency "${{ inputs.warp_concurrency || '64' }}" \
--log-file /tmp/rustfs-perf-test.log
- name: Analyze results
if: ${{ steps.benchmark.conclusion == 'success' }}
run: |
./auto-testing/rustfs_performance_test.sh --step 6 -y
- name: Post results to backlog issue
if: ${{ steps.benchmark.conclusion == 'success' }}
env:
GH_TOKEN: ${{ env.PF_TESTING_GH_TOKEN }}
RESULT_DIR: ${{ env.RUSTFS_RESULT_DIR }}
run: |
set -euo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; skipping issue post"
exit 0
fi
SUMMARY="${RESULT_DIR}/summary.md"
[ -f "${SUMMARY}" ] || { echo "summary.md not found at ${SUMMARY}"; exit 1; }
DATE="$(date -u +%Y-%m-%d)"
{
echo "## RustFS nightly build performance testing report"
echo ""
echo "- **日期**: ${DATE}"
echo "- **Run**: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
echo "- **触发方式**: ${{ github.event_name }}"
echo ""
cat "${SUMMARY}"
} > /tmp/rustfs-perf-issue-body.md
TITLE="RustFS nightly build performance testing report"
EXISTING="$(gh issue list --repo rustfs/backlog \
--search "in:title \"${TITLE}\"" --state all --limit 5 \
--json number --jq '.[0].number // empty')"
if [ -n "${EXISTING}" ]; then
gh issue comment "${EXISTING}" --repo rustfs/backlog --body-file /tmp/rustfs-perf-issue-body.md
echo "commented on existing issue #${EXISTING}"
else
gh issue create --repo rustfs/backlog --title "${TITLE}" --body-file /tmp/rustfs-perf-issue-body.md
fi
- name: Upload test logs & results
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: rustfs-perf-test-${{ github.run_id }}
path: |
/tmp/rustfs-perf-test*.log
/tmp/rustfs-perf-results/**
if-no-files-found: warn
- name: Reset test environment (after)
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
./auto-testing/rustfs_performance_test.sh --step 7 -y
- name: Notify on failure
if: failure()
run: |
echo "RustFS performance test failed"
echo "Package source: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
echo "See the uploaded log artifact for details."
+266 -20
View File
@@ -1,4 +1,4 @@
name: RustFS Pool Expansion / Decommission Test
name: RustFS Functional Test Suite (S3/KMS/tier/pool/heal)
on:
workflow_dispatch:
@@ -30,6 +30,14 @@ on:
description: 'Run the pool decommission step (3-pool topology only)'
type: boolean
default: true
stop_node_gb:
description: 'Heal: stop the outage node when surviving nodes reach N GiB'
required: false
default: '15'
warp_stop_gb:
description: 'Heal: stop warp when surviving nodes reach N GiB'
required: false
default: '40'
cleanup_before:
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
type: boolean
@@ -38,15 +46,17 @@ on:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
schedule:
# Nightly regression run; remove if you do not want a schedule.
- cron: '0 21 * * *'
workflow_run:
# Run after the nightly build completes: S3 -> KMS -> tier -> pool -> heal.
workflows: ["Nightly GNU Build"]
types: [completed]
permissions:
contents: read
# Only one pool-expansion test at a time: the workflow mutates a shared
# test environment, so concurrent runs must not clobber each other.
# Only one test run at a time: every job mutates the same shared test
# environment (vm000/vm001/vm002), so concurrent runs must not clobber each
# other. Jobs inside a run are chained with needs to serialize them.
concurrency:
group: rustfs-pool-expansion-test
cancel-in-progress: false
@@ -61,19 +71,181 @@ env:
RUSTFS_API_ENDPOINT: ${{ secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }}
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
# Package used by the scheduled run (workflow_dispatch inputs are empty for
# schedule events), i.e. the latest nightly deb published by nightly-gnu.yml.
# Package used by the nightly run (workflow_dispatch inputs are empty for
# workflow_run events), i.e. the latest nightly deb published by nightly-gnu.yml.
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
jobs:
pool-expansion-test:
# All test scripts live in rustfs/auto-testing; rustfs stores only this
# workflow. Each job checks out auto-testing before running.
s3-compat-test:
name: S3 compatibility test
runs-on: smoke-testing
timeout-minutes: 360
timeout-minutes: 240
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
steps:
- name: Checkout
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/auto-testing
ref: main
path: auto-testing
persist-credentials: false
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Run S3 compatibility suite
run: |
chmod +x auto-testing/rustfs-s3-compat-test.sh
ARGS=(--all-topologies -y --log-file /tmp/rustfs-s3-compat.log)
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
elif [ -n "${{ inputs.rustfs_version }}" ]; then
ARGS+=(--version "${{ inputs.rustfs_version }}")
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./auto-testing/rustfs-s3-compat-test.sh "${ARGS[@]}"
- name: Upload test logs
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: rustfs-s3-compat-${{ github.run_id }}
path: |
/tmp/rustfs-s3-compat*.log
if-no-files-found: warn
- name: Notify on failure
if: failure()
run: |
echo "RustFS S3 compatibility suite failed"
echo "See the uploaded log artifact for details."
kms-test:
name: KMS test (after S3)
runs-on: smoke-testing
timeout-minutes: 360
needs: s3-compat-test
if: ${{ always() && (github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success') }}
steps:
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/auto-testing
ref: main
path: auto-testing
persist-credentials: false
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Ensure docker (Vault container)
run: |
if ! command -v docker >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y docker.io
fi
sudo systemctl enable --now docker
docker info >/dev/null 2>&1 || sudo docker info >/dev/null 2>&1
- name: Run KMS suite
run: |
chmod +x auto-testing/rustfs-kms-test.sh
ARGS=(--all-topologies --backends local,vault-kv2 -y --log-file /tmp/rustfs-kms.log)
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
elif [ -n "${{ inputs.rustfs_version }}" ]; then
ARGS+=(--version "${{ inputs.rustfs_version }}")
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./auto-testing/rustfs-kms-test.sh "${ARGS[@]}"
- name: Upload test logs
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: rustfs-kms-test-${{ github.run_id }}
path: |
/tmp/rustfs-kms*.log
if-no-files-found: warn
- name: Notify on failure
if: failure()
run: |
echo "RustFS KMS suite failed"
echo "See the uploaded log artifact for details."
tier-test:
name: Tier / event / audit test (after KMS)
runs-on: smoke-testing
timeout-minutes: 360
needs: kms-test
if: ${{ always() && (github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success') }}
steps:
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/auto-testing
ref: main
path: auto-testing
persist-credentials: false
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Ensure MQTT broker + clients (event notification tests)
run: |
if ! command -v mosquitto_sub >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y mosquitto mosquitto-clients
fi
sudo mkdir -p /etc/mosquitto/conf.d
printf 'listener 1883 0.0.0.0\nallow_anonymous true\n' | sudo tee /etc/mosquitto/conf.d/rustfs-test.conf >/dev/null
sudo systemctl restart mosquitto
sleep 2
ss -tln 2>/dev/null | grep -q ':1883' || { echo "mosquitto not listening on 1883"; exit 1; }
- name: Run tier / event / audit suite
run: |
chmod +x auto-testing/rustfs-tier-test.sh
ARGS=(--all-topologies -y --log-file /tmp/rustfs-tier.log)
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
elif [ -n "${{ inputs.rustfs_version }}" ]; then
ARGS+=(--version "${{ inputs.rustfs_version }}")
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./auto-testing/rustfs-tier-test.sh "${ARGS[@]}"
- name: Upload test logs
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: rustfs-tier-test-${{ github.run_id }}
path: |
/tmp/rustfs-tier*.log
if-no-files-found: warn
- name: Notify on failure
if: failure()
run: |
echo "RustFS tier/event/audit suite failed"
echo "See the uploaded log artifact for details."
pool-expansion-test:
name: Pool expansion / decommission test (after tier)
runs-on: smoke-testing
timeout-minutes: 360
needs: tier-test
if: ${{ always() && (github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success') }}
steps:
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/auto-testing
ref: main
path: auto-testing
persist-credentials: false
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Show environment
run: |
@@ -86,12 +258,12 @@ jobs:
- name: Reset test environment (before)
if: ${{ inputs.cleanup_before != 'false' }}
run: |
chmod +x scripts/test/rustfs_pool_expand.sh
./scripts/test/rustfs_pool_expand.sh --reset -y
chmod +x auto-testing/rustfs_pool_expand.sh
./auto-testing/rustfs_pool_expand.sh --reset -y
- name: Install RustFS package & start first pool
run: |
ARGS=(--steps 1,2,3 -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
ARGS=(--steps "1,2,3" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
elif [ -n "${{ inputs.rustfs_version }}" ]; then
@@ -99,7 +271,7 @@ jobs:
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./scripts/test/rustfs_pool_expand.sh "${ARGS[@]}"
./auto-testing/rustfs_pool_expand.sh "${ARGS[@]}"
- name: Preflight checks
run: |
@@ -111,7 +283,7 @@ jobs:
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./scripts/test/rustfs_pool_expand.sh "${ARGS[@]}"
./auto-testing/rustfs_pool_expand.sh "${ARGS[@]}"
- name: Run pool expansion & decommission test
id: pool_test
@@ -124,7 +296,7 @@ jobs:
STEPS="$STEPS,9"
fi
fi
./scripts/test/rustfs_pool_expand.sh \
./auto-testing/rustfs_pool_expand.sh \
--steps "$STEPS" --with-warp -y \
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
--storage-threshold "${{ inputs.storage_threshold || '50' }}" \
@@ -137,14 +309,14 @@ jobs:
with:
name: rustfs-pool-test-${{ github.run_id }}
path: |
/tmp/rustfs-pool-test.log
/tmp/rustfs-warp.log
/tmp/rustfs-pool-test*.log
/tmp/rustfs-warp.*.log
if-no-files-found: warn
- name: Reset test environment (after)
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
./scripts/test/rustfs_pool_expand.sh --reset -y
./auto-testing/rustfs_pool_expand.sh --reset -y
- name: Notify on failure
if: failure()
@@ -152,3 +324,77 @@ jobs:
echo "RustFS pool expansion test failed"
echo "Package source: ${{ inputs.package_url || inputs.rustfs_version || 'nightly (R2 latest)' }}"
echo "See the uploaded log artifact for details."
# Heal regression runs after the pool test regardless of its outcome: a pool
# failure must be reported (it makes the run red) but must not block heal.
heal-test:
name: Heal test (after pool test)
runs-on: smoke-testing
timeout-minutes: 480
needs: pool-expansion-test
if: ${{ always() && (github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success') }}
steps:
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/auto-testing
ref: main
path: auto-testing
persist-credentials: false
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Reset test environment (before)
if: ${{ inputs.cleanup_before != 'false' }}
run: |
chmod +x auto-testing/rustfs_heal_test.sh
./auto-testing/rustfs_heal_test.sh --reset -y
- name: Install RustFS package & start cluster
run: |
ARGS=(--steps "1,2" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}"
- name: Preflight checks
run: |
ARGS=(--preflight --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}"
- name: Run heal test (write -> outage -> heal -> verify)
run: |
./auto-testing/rustfs_heal_test.sh \
--steps 3,4,5,6,7 -y \
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
--stop-node-gb "${{ inputs.stop_node_gb || '15' }}" \
--warp-stop-gb "${{ inputs.warp_stop_gb || '40' }}" \
--log-file /tmp/rustfs-heal-test.log
- name: Upload test logs
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: rustfs-heal-test-${{ github.run_id }}
path: |
/tmp/rustfs-heal-test.log
/tmp/rustfs-warp.*.log
if-no-files-found: warn
- name: Reset test environment (after)
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
./auto-testing/rustfs_heal_test.sh --reset -y
- name: Notify on failure
if: failure()
run: |
echo "RustFS heal test failed"
echo "See the uploaded log artifact for details."
+7 -2
View File
@@ -31,8 +31,13 @@ This file contains repository-wide rules. Use the nearest subdirectory
- An existing clean, isolated task worktree is sufficient. Create another
worktree only when the current checkout is shared, dirty with unrelated work,
or belongs to another task.
- Never commit from a shared checkout. Use an `overtrue/` feature branch unless
the user requests another name.
- Never commit from a shared checkout.
- Use a task-specific branch named `<type>/<topic>`, such as `fix/...`,
`feat/...`, `test/...`, or `docs/...`, unless the user specifies a name.
- Do not include agent, tool, contributor, account, or organization names in
branch names.
- Push to the user-requested remote or the repository's configured push remote.
Do not hard-code or infer a remote from an account name.
- Check free space before artifact-heavy builds, tests, coverage, or downloads.
Re-check before a broad gate when space is tight.
- Remove only task-owned temporary/build artifacts. Never delete another task's
Generated
+62 -44
View File
@@ -62,7 +62,7 @@ checksum = "f8eb277bec05f56a0e0591f155a484cbd0f4f07ff2905051a48c72f004f7ed58"
dependencies = [
"cipher 0.5.2",
"cpubits",
"cpufeatures 0.3.0",
"cpufeatures 0.3.1",
"zeroize",
]
@@ -323,13 +323,13 @@ checksum = "03918c3dbd7701a85c6b9887732e2921175f26c350b4563841d0958c21d57e6d"
[[package]]
name = "argon2"
version = "0.6.0-rc.8"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7af50940b73bf4e16c15c448a2b121c63f2d68e3e54b6a8731673cb4aa0cdff5"
checksum = "134c52ddac6d63c576bef8168db10c83c49c26444ecbc68060fef078925a901c"
dependencies = [
"base64ct",
"blake2",
"cpufeatures 0.3.0",
"cpufeatures 0.3.1",
"password-hash",
]
@@ -1024,7 +1024,7 @@ dependencies = [
"http 0.2.12",
"http 1.5.0",
"http-body 1.1.0",
"lru 0.18.2",
"lru 0.18.3",
"percent-encoding",
"regex-lite",
"sha2 0.11.0",
@@ -1939,13 +1939,13 @@ checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527"
[[package]]
name = "chacha20"
version = "0.10.1"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81"
checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06"
dependencies = [
"cfg-if",
"cipher 0.5.2",
"cpufeatures 0.3.0",
"cpufeatures 0.3.1",
"rand_core 0.10.1",
"zeroize",
]
@@ -2283,9 +2283,9 @@ dependencies = [
[[package]]
name = "convert_case"
version = "0.11.0"
version = "0.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "affbf0190ed2caf063e3def54ff444b449371d55c58e513a95ab98eca50adb49"
checksum = "1af709f1f33454bf52eadfc8c78b3b9ef9cb26fb54d16dc9cd9a7299f899fd1b"
dependencies = [
"unicode-segmentation",
]
@@ -2348,9 +2348,9 @@ dependencies = [
[[package]]
name = "cpufeatures"
version = "0.3.0"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201"
checksum = "5ca28b0ae3115b884660db4118d803791fd6756b6e88f39c0f3f7859060d7566"
dependencies = [
"libc",
]
@@ -2611,7 +2611,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b5eed333089e2e1c1ac8c6c0398e5e2497b4c9926ca6d0365ed1e099afa5bc23"
dependencies = [
"cfg-if",
"cpufeatures 0.3.0",
"cpufeatures 0.3.1",
"curve25519-dalek-derive",
"digest 0.11.3",
"fiat-crypto 0.3.0",
@@ -5885,7 +5885,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d8f198d1db720e4940b5a493201d199d9f24f568f8f746bd13706243a2f71598"
dependencies = [
"cfg-if",
"cpufeatures 0.3.0",
"cpufeatures 0.3.1",
]
[[package]]
@@ -6103,9 +6103,9 @@ checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981"
[[package]]
name = "libredox"
version = "0.1.20"
version = "0.1.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "28d0a00925a9f930d679b6789b721e3a7f9ed110f41b86d2497caa780c3a070a"
checksum = "d7955dfc218a8afb29dfeffd540e3a6e96baeb94fe7138228dd7cc6937fbbf96"
dependencies = [
"libc",
]
@@ -6209,9 +6209,9 @@ dependencies = [
[[package]]
name = "lru"
version = "0.18.2"
version = "0.18.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d2f2f9b4ba7e6b24d95e7e899329d35be83bcded72c8540cdd5368932d1d90a"
checksum = "0d317b4b9eb398e6acce275758ec6125535505e7a146fb1a9b8bda2451b0ff4c"
dependencies = [
"hashbrown 0.17.1",
]
@@ -6659,7 +6659,7 @@ dependencies = [
"futures-sink",
"futures-util",
"keyed_priority_queue",
"lru 0.18.2",
"lru 0.18.3",
"mysql_common",
"percent-encoding",
"rand 0.10.2",
@@ -7371,9 +7371,9 @@ checksum = "1a80800c0488c3a21695ea981a54918fbb37abf04f4d0720c453632255e2ff0e"
[[package]]
name = "owo-colors"
version = "4.3.0"
version = "4.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d211803b9b6b570f68772237e415a029d5a50c65d382910b879fb19d3271f94d"
checksum = "13c45bb4a6ae1280ec0803b1ef9d3455eb50f01efbbe1447ab020f1d54fba9d8"
[[package]]
name = "p12-keystore"
@@ -7924,7 +7924,7 @@ version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e2d0073b297041425c7c3df6eb4792d598a15323fe63346852b092eca02904c"
dependencies = [
"cpufeatures 0.3.0",
"cpufeatures 0.3.1",
"universal-hash",
"zeroize",
]
@@ -7936,7 +7936,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0fa31d631f2b2cb2a544d0aa321ce847a94764d701ca2becc411138b93d49cd"
dependencies = [
"cpubits",
"cpufeatures 0.3.0",
"cpufeatures 0.3.1",
"universal-hash",
"zeroize",
]
@@ -8512,16 +8512,6 @@ version = "1.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0"
[[package]]
name = "quick-xml"
version = "0.41.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e660451e55124f798a69a5af3f49ccfbefbd41910eefd25caf2393e1f3473ec1"
dependencies = [
"memchr",
"serde",
]
[[package]]
name = "quick-xml"
version = "0.42.0"
@@ -9414,7 +9404,7 @@ dependencies = [
"percent-encoding",
"pin-project-lite",
"proptest",
"quick-xml 0.42.0",
"quick-xml",
"rand 0.10.2",
"rcgen",
"regex",
@@ -10157,7 +10147,7 @@ dependencies = [
"jiff",
"metrics",
"percent-encoding",
"quick-xml 0.42.0",
"quick-xml",
"rayon",
"rustc-hash",
"rustfs-config",
@@ -10489,7 +10479,7 @@ dependencies = [
"hyper-util",
"lazy_static",
"md-5 0.11.0",
"quick-xml 0.42.0",
"quick-xml",
"rand 0.10.2",
"rustfs-checksums",
"rustfs-config",
@@ -10611,6 +10601,7 @@ dependencies = [
"s3s",
"serde",
"serde_json",
"serial_test",
"sha2 0.11.0",
"temp-env",
"tempfile",
@@ -10815,7 +10806,7 @@ dependencies = [
"blake2",
"brotli",
"bytes",
"convert_case 0.11.0",
"convert_case 0.12.0",
"crc-fast",
"criterion",
"flate2",
@@ -11042,7 +11033,7 @@ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
[[package]]
name = "s3s"
version = "0.15.0"
source = "git+https://github.com/rustfs/s3s.git?rev=0f6f83d98b37fd9edcaa3be573db4aa8f568e088#0f6f83d98b37fd9edcaa3be573db4aa8f568e088"
source = "git+https://github.com/rustfs/s3s.git?rev=6e7b41252c7ba218a90886f58d297716ddf68acf#6e7b41252c7ba218a90886f58d297716ddf68acf"
dependencies = [
"arc-swap",
"arrayvec",
@@ -11063,14 +11054,17 @@ dependencies = [
"httparse",
"hyper",
"itoa",
"jiff",
"md-5 0.11.0",
"memchr",
"mime",
"nom 8.0.0",
"numeric_cast",
"pin-project-lite",
"quick-xml 0.41.0",
"quick-xml",
"regex",
"s3s-sigv2",
"s3s-sigv4",
"serde",
"serde_json",
"serde_urlencoded",
@@ -11092,6 +11086,30 @@ dependencies = [
"zeroize",
]
[[package]]
name = "s3s-sigv2"
version = "0.16.0-alpha.1"
source = "git+https://github.com/rustfs/s3s.git?rev=6e7b41252c7ba218a90886f58d297716ddf68acf#6e7b41252c7ba218a90886f58d297716ddf68acf"
dependencies = [
"jiff",
"thiserror 2.0.20",
]
[[package]]
name = "s3s-sigv4"
version = "0.16.0-alpha.1"
source = "git+https://github.com/rustfs/s3s.git?rev=6e7b41252c7ba218a90886f58d297716ddf68acf#6e7b41252c7ba218a90886f58d297716ddf68acf"
dependencies = [
"arrayvec",
"base64-simd",
"hex-simd",
"jiff",
"nom 8.0.0",
"serde",
"smallvec",
"thiserror 2.0.20",
]
[[package]]
name = "salsa20"
version = "0.10.2"
@@ -11473,7 +11491,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214"
dependencies = [
"cfg-if",
"cpufeatures 0.3.0",
"cpufeatures 0.3.1",
"digest 0.11.3",
]
@@ -11501,7 +11519,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4"
dependencies = [
"cfg-if",
"cpufeatures 0.3.0",
"cpufeatures 0.3.1",
"digest 0.11.3",
]
@@ -12174,7 +12192,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom 0.4.3",
"getrandom 0.3.4",
"once_cell",
"rustix",
"windows-sys 0.61.2",
@@ -12864,9 +12882,9 @@ dependencies = [
[[package]]
name = "twox-hash"
version = "2.1.3"
version = "2.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8464ec13c3691491391d9fce00f6416c9a48e46972f72d7865688be2080192c9"
checksum = "5283634e518fe9e82c7b20520bb4bc209009fd16c82077c802f8111ecbb0117a"
[[package]]
name = "typed-path"
+3 -3
View File
@@ -198,7 +198,7 @@ serde_urlencoded = "0.7.1"
# have incompatible APIs. Keep them exact-pinned and monitor upstream for stable
# releases.
aes-gcm = { version = "=0.11.1" }
argon2 = { version = "=0.6.0-rc.8" }
argon2 = { version = "=0.6.0" }
blake2 = "=0.11.0"
chacha20poly1305 = { version = "=0.11.0" }
crc-fast = "1.10.0"
@@ -247,7 +247,7 @@ base64-simd = "0.8.0"
brotli = "8.0.4"
clap = { version = "4.6.6" }
const-str = { version = "1.1.0" }
convert_case = "0.11.0"
convert_case = "0.12.0"
criterion = { version = "0.8" }
crossbeam-queue = "0.3.13"
crossbeam-channel = "0.5.16"
@@ -304,7 +304,7 @@ rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.4" }
rust-embed = { version = "8.12.0" }
rustc-hash = { version = "2.1.3" }
s3s = { git = "https://github.com/rustfs/s3s.git", rev = "0f6f83d98b37fd9edcaa3be573db4aa8f568e088", version = "0.15.0", features = ["minio"] }
s3s = { git = "https://github.com/rustfs/s3s.git", rev = "6e7b41252c7ba218a90886f58d297716ddf68acf", version = "0.15.0", features = ["minio"] }
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.3"
+1 -1
View File
@@ -16,7 +16,7 @@
</p>
<p align="center">
<a href="https://docs.rustfs.com/installation/">Getting Started</a>
<a href="https://docs.rustfs.com/en/installation">Getting Started</a>
· <a href="https://docs.rustfs.com/">Docs</a>
· <a href="https://github.com/rustfs/rustfs/issues">Bug reports</a>
· <a href="https://github.com/rustfs/rustfs/discussions">Discussions</a>
+1 -1
View File
@@ -16,7 +16,7 @@
</p>
<p align="center">
<a href="https://docs.rustfs.com/installation/">快速开始</a>
<a href="https://docs.rustfs.com/zh/installation">快速开始</a>
· <a href="https://docs.rustfs.com/">文档</a>
· <a href="https://github.com/rustfs/rustfs/issues">报告 Bug</a>
· <a href="https://github.com/rustfs/rustfs/discussions">社区讨论</a>
+271
View File
@@ -178,6 +178,76 @@ pub trait WorkloadAdmissionSnapshotProvider {
fn workload_admission_snapshot(&self) -> WorkloadAdmissionRegistrySnapshot;
}
/// Foreground workload pressure observed against a configured utilization threshold.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ForegroundPressure {
/// Foreground workload class whose utilization reached its threshold.
pub class: WorkloadClass,
/// Observed utilization percentage for the class.
pub usage_pct: usize,
/// Configured threshold percentage that the observed utilization reached.
pub threshold_pct: usize,
}
impl ForegroundPressure {
/// Return a stable reason label for logs and metrics.
pub const fn reason(self) -> &'static str {
match self.class {
WorkloadClass::ForegroundRead => "foreground_read_pressure",
WorkloadClass::ForegroundWrite => "foreground_write_pressure",
_ => "foreground_pressure",
}
}
}
/// Return the strongest foreground pressure in `snapshot`, if any.
///
/// A zero threshold disables its class. `Saturated` counts as full utilization
/// regardless of the reported limit; otherwise a class contributes only when it
/// reports a non-zero limit, with a missing active count read as zero. When both
/// classes are above their threshold the higher utilization wins.
///
/// Callers own the enable switch: this function evaluates thresholds only.
pub fn foreground_pressure(
snapshot: &WorkloadAdmissionRegistrySnapshot,
read_threshold_pct: usize,
write_threshold_pct: usize,
) -> Option<ForegroundPressure> {
[
(WorkloadClass::ForegroundRead, read_threshold_pct),
(WorkloadClass::ForegroundWrite, write_threshold_pct),
]
.into_iter()
.filter_map(|(class, threshold_pct)| {
if threshold_pct == 0 {
return None;
}
let entry = snapshot.get(class)?;
let usage_pct = if matches!(entry.state, AdmissionState::Saturated) {
100
} else {
let limit = entry.limit?;
if limit == 0 {
return None;
}
entry
.active
.unwrap_or(0)
.saturating_mul(100)
.checked_div(limit)
.unwrap_or(100)
};
(usage_pct >= threshold_pct).then_some(ForegroundPressure {
class,
usage_pct,
threshold_pct,
})
})
.max_by_key(|pressure| pressure.usage_pct)
}
#[cfg(test)]
mod tests {
use super::*;
@@ -314,4 +384,205 @@ mod tests {
assert!(err.to_string().contains("unexpected"));
}
fn counted(
class: WorkloadClass,
state: AdmissionState,
active: Option<usize>,
limit: Option<usize>,
) -> WorkloadAdmissionSnapshot {
WorkloadAdmissionSnapshot::new(class, state).with_counts(active, None, limit)
}
fn registry(entries: Vec<WorkloadAdmissionSnapshot>) -> WorkloadAdmissionRegistrySnapshot {
WorkloadAdmissionRegistrySnapshot::new(entries)
}
#[test]
fn foreground_pressure_reason_labels_cover_non_foreground_classes() {
let read = ForegroundPressure {
class: WorkloadClass::ForegroundRead,
usage_pct: 90,
threshold_pct: 80,
};
let write = ForegroundPressure {
class: WorkloadClass::ForegroundWrite,
usage_pct: 90,
threshold_pct: 80,
};
let repair = ForegroundPressure {
class: WorkloadClass::Repair,
usage_pct: 90,
threshold_pct: 80,
};
assert_eq!(read.reason(), "foreground_read_pressure");
assert_eq!(write.reason(), "foreground_write_pressure");
assert_eq!(repair.reason(), "foreground_pressure");
}
#[test]
fn foreground_pressure_is_disabled_when_both_thresholds_are_zero() {
let snapshot = registry(vec![
counted(WorkloadClass::ForegroundRead, AdmissionState::Saturated, Some(8), Some(8)),
counted(WorkloadClass::ForegroundWrite, AdmissionState::Saturated, Some(8), Some(8)),
]);
assert_eq!(foreground_pressure(&snapshot, 0, 0), None);
}
#[test]
fn foreground_pressure_skips_only_the_class_whose_threshold_is_zero() {
let snapshot = registry(vec![
counted(WorkloadClass::ForegroundRead, AdmissionState::Open, Some(10), Some(10)),
counted(WorkloadClass::ForegroundWrite, AdmissionState::Open, Some(9), Some(10)),
]);
assert_eq!(
foreground_pressure(&snapshot, 0, 80),
Some(ForegroundPressure {
class: WorkloadClass::ForegroundWrite,
usage_pct: 90,
threshold_pct: 80,
})
);
assert_eq!(
foreground_pressure(&snapshot, 80, 0),
Some(ForegroundPressure {
class: WorkloadClass::ForegroundRead,
usage_pct: 100,
threshold_pct: 80,
})
);
}
#[test]
fn foreground_pressure_ignores_missing_entries() {
let snapshot = registry(vec![counted(WorkloadClass::Scanner, AdmissionState::Saturated, Some(8), Some(8))]);
assert_eq!(foreground_pressure(&snapshot, 1, 1), None);
}
#[test]
fn foreground_pressure_ignores_missing_and_zero_limits() {
let missing_limit = registry(vec![counted(
WorkloadClass::ForegroundRead,
AdmissionState::Throttled,
Some(8),
None,
)]);
let zero_limit = registry(vec![counted(
WorkloadClass::ForegroundWrite,
AdmissionState::Throttled,
Some(8),
Some(0),
)]);
assert_eq!(foreground_pressure(&missing_limit, 1, 1), None);
assert_eq!(foreground_pressure(&zero_limit, 1, 1), None);
}
#[test]
fn foreground_pressure_treats_saturated_as_full_without_reading_limit() {
let snapshot = registry(vec![
counted(WorkloadClass::ForegroundRead, AdmissionState::Saturated, None, None),
counted(WorkloadClass::ForegroundWrite, AdmissionState::Saturated, Some(0), Some(0)),
]);
assert_eq!(
foreground_pressure(&snapshot, 100, 0),
Some(ForegroundPressure {
class: WorkloadClass::ForegroundRead,
usage_pct: 100,
threshold_pct: 100,
})
);
assert_eq!(
foreground_pressure(&snapshot, 0, 100),
Some(ForegroundPressure {
class: WorkloadClass::ForegroundWrite,
usage_pct: 100,
threshold_pct: 100,
})
);
}
#[test]
fn foreground_pressure_reads_missing_active_as_zero() {
let snapshot = registry(vec![counted(WorkloadClass::ForegroundRead, AdmissionState::Open, None, Some(8))]);
assert_eq!(foreground_pressure(&snapshot, 1, 1), None);
}
#[test]
fn foreground_pressure_returns_the_higher_utilization_when_both_classes_exceed() {
let read_higher = registry(vec![
counted(WorkloadClass::ForegroundRead, AdmissionState::Open, Some(19), Some(20)),
counted(WorkloadClass::ForegroundWrite, AdmissionState::Open, Some(17), Some(20)),
]);
let write_higher = registry(vec![
counted(WorkloadClass::ForegroundRead, AdmissionState::Open, Some(17), Some(20)),
counted(WorkloadClass::ForegroundWrite, AdmissionState::Open, Some(19), Some(20)),
]);
assert_eq!(
foreground_pressure(&read_higher, 80, 80),
Some(ForegroundPressure {
class: WorkloadClass::ForegroundRead,
usage_pct: 95,
threshold_pct: 80,
})
);
assert_eq!(
foreground_pressure(&write_higher, 80, 80),
Some(ForegroundPressure {
class: WorkloadClass::ForegroundWrite,
usage_pct: 95,
threshold_pct: 80,
})
);
}
#[test]
fn foreground_pressure_breaks_utilization_ties_toward_the_write_class() {
let snapshot = registry(vec![
counted(WorkloadClass::ForegroundRead, AdmissionState::Open, Some(18), Some(20)),
counted(WorkloadClass::ForegroundWrite, AdmissionState::Open, Some(18), Some(20)),
]);
assert_eq!(
foreground_pressure(&snapshot, 80, 80),
Some(ForegroundPressure {
class: WorkloadClass::ForegroundWrite,
usage_pct: 90,
threshold_pct: 80,
})
);
}
#[test]
fn foreground_pressure_triggers_exactly_at_the_threshold_and_not_below() {
let at_threshold = registry(vec![counted(
WorkloadClass::ForegroundRead,
AdmissionState::Open,
Some(8),
Some(10),
)]);
let below_threshold = registry(vec![counted(
WorkloadClass::ForegroundRead,
AdmissionState::Open,
Some(7),
Some(10),
)]);
assert_eq!(
foreground_pressure(&at_threshold, 80, 80),
Some(ForegroundPressure {
class: WorkloadClass::ForegroundRead,
usage_pct: 80,
threshold_pct: 80,
})
);
assert_eq!(foreground_pressure(&below_threshold, 80, 80), None);
}
}
+8
View File
@@ -40,6 +40,14 @@ pub const HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS: u64 = 5_000;
pub const ENV_HEALTH_CLUSTER_TIMEOUT_MS: &str = "RUSTFS_HEALTH_CLUSTER_TIMEOUT_MS";
pub const DEFAULT_HEALTH_CLUSTER_TIMEOUT_MS: u64 = 2000;
/// Timeout for one remote lock-client online check used by readiness (milliseconds).
///
/// This is intentionally shorter than the generic lock RPC timeout so
/// `/health/ready` can report degradation instead of riding a dead peer's
/// connect or HTTP/2 keepalive budget.
pub const ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS: &str = "RUSTFS_HEALTH_LOCK_ONLINE_TIMEOUT_MS";
pub const DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS: u64 = 1000;
/// Maximum time to wait for local node runtime readiness (storage / IAM / lock
/// quorum) during startup before failing fast (seconds).
///
+43
View File
@@ -288,6 +288,49 @@ pub const DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 0;
const _: () = assert!(!DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE);
/// Enable automatic foreground admission for large or unknown-size PutObject requests.
///
/// Unlike the strict experimental gate above, this default-on path only applies
/// to requests that are large enough to create sustained erasure/RPC pressure.
/// Small PUTs continue on the legacy path unless the strict gate is explicitly
/// enabled.
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE";
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: bool = true;
/// Maximum automatic foreground write requests admitted concurrently per process.
///
/// `0` derives a conservative default from the local disk-read scheduler cap,
/// currently clamped to protect the commit path without making ordinary high
/// throughput uploads single-file.
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT";
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: usize = 0;
/// Minimum direct PutObject size that enters automatic foreground write admission.
///
/// Requests with an unknown size are treated as large because the write pressure
/// cannot be bounded from headers.
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES";
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 32 * 1024 * 1024;
/// Minimum UploadPart size that enters automatic foreground write admission.
///
/// Multipart pressure is often many moderate-sized parts rather than one very
/// large request. The default gates every multipart part through the same permit
/// pool as large/unknown-size PutObject while keeping small direct PUTs on the
/// legacy path.
pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: &str =
"RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES";
pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 0;
/// Time in milliseconds an automatic foreground write waits for a permit.
///
/// A short wait smooths transient bursts while still returning S3
/// `SlowDown`/503 before body ingest when the node is already saturated.
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 250;
const _: () = assert!(DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE);
/// Environment variable for minimum GetObject timeout in seconds.
///
/// When dynamic timeout calculation is enabled, this is the minimum timeout
+9 -53
View File
@@ -31,14 +31,9 @@
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client, rustfs_binary_path};
use aws_sdk_s3::config::{Credentials, Region};
use crate::common::{RustFSTestEnvironment, init_logging, rustfs_binary_path};
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::{Client, Config};
use http::header::HOST;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use std::error::Error;
use std::io::Read;
use std::process::{Command, Stdio};
@@ -87,10 +82,10 @@ mod tests {
}
/// Send a SigV4-signed request to `path` (optionally with a JSON `body`) and
/// return `(status, body)`. Uses the `UNSIGNED_PAYLOAD` content hash so a
/// request body can be attached without the caller pre-hashing it — the
/// server verifies the signature against the same sentinel, exactly as the
/// AWS SDKs / MinIO client do for streaming/unsigned payloads.
/// return `(status, body)`.
///
/// Thin wrapper over [`crate::common::admin_request`], kept local so the
/// call sites below keep their `Option<&str>` body shape.
async fn signed_request(
base_url: &str,
method: http::Method,
@@ -99,47 +94,13 @@ mod tests {
access_key: &str,
secret_key: &str,
) -> Result<(reqwest::StatusCode, String), Box<dyn Error + Send + Sync>> {
let url = format!("{base_url}{path}");
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("missing authority")?.to_string();
let body_bytes = body.map(|b| b.as_bytes().to_vec()).unwrap_or_default();
// The signature is computed over `UNSIGNED_PAYLOAD`, so the body bytes do
// not participate in the SigV4 hash — sign over an empty body and attach
// the real payload to the wire request below.
let request = http::Request::builder()
.method(method.clone())
.uri(uri)
.header(HOST, authority)
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
let client = local_http_client();
let mut rb = client.request(method, url.as_str());
for (name, value) in signed.headers() {
rb = rb.header(name, value);
}
if !body_bytes.is_empty() {
rb = rb.body(body_bytes);
}
let resp = rb.send().await?;
let status = resp.status();
let text = resp.text().await?;
Ok((status, text))
crate::common::admin_request(base_url, method, path, body.map(str::to_string), access_key, secret_key).await
}
/// Build an S3 client bound to explicit credentials (used to exercise the S3
/// data plane with rotated / stale root credentials).
fn s3_client_with(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
let credentials = Credentials::new(access_key, secret_key, None, None, "sec4-admin-auth");
let config = Config::builder()
.credentials_provider(credentials)
.region(Region::new("us-east-1"))
.endpoint_url(&env.url)
.force_path_style(true)
.behavior_version_latest()
.build();
Client::from_conf(config)
env.create_s3_client_with_credentials(access_key, secret_key)
}
/// Create a non-admin IAM user via the admin `add-user` API using the root
@@ -151,12 +112,7 @@ mod tests {
access_key: &str,
secret_key: &str,
) -> Result<(), Box<dyn Error + Send + Sync>> {
let path = format!("/rustfs/admin/v3/add-user?accessKey={access_key}");
let body = serde_json::json!({ "secretKey": secret_key, "status": "enabled" }).to_string();
let (status, resp) =
signed_request(&env.url, http::Method::PUT, &path, Some(&body), &env.access_key, &env.secret_key).await?;
assert!(status.is_success(), "add-user should succeed (status={status}, body={resp})");
Ok(())
crate::common::admin_create_user(env, access_key, secret_key).await
}
/// A fully authenticated but non-admin credential must be rejected with
+3 -26
View File
@@ -59,8 +59,8 @@ mod tests {
/// One signed admin request, returning the status and the raw body.
///
/// Signs with `UNSIGNED_PAYLOAD` so the body does not participate in the
/// hash, matching how the other admin e2e tests drive these routes.
/// Thin wrapper over [`crate::common::admin_request`], kept local so the
/// call sites below keep their `Option<&str>` body shape.
async fn signed_request(
base_url: &str,
method: http::Method,
@@ -69,30 +69,7 @@ mod tests {
access_key: &str,
secret_key: &str,
) -> Result<(reqwest::StatusCode, String), Box<dyn Error + Send + Sync>> {
let url = format!("{base_url}{path}");
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("missing authority")?.to_string();
let body_bytes = body.map(|b| b.as_bytes().to_vec()).unwrap_or_default();
let request = http::Request::builder()
.method(method.clone())
.uri(uri)
.header(HOST, authority)
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
let client = local_http_client();
let mut builder = client.request(method, url.as_str());
for (name, value) in signed.headers() {
builder = builder.header(name, value);
}
if !body_bytes.is_empty() {
builder = builder.body(body_bytes);
}
let response = builder.send().await?;
let status = response.status();
let text = response.text().await?;
Ok((status, text))
crate::common::admin_request(base_url, method, path, body.map(str::to_string), access_key, secret_key).await
}
/// A SigV4-signed `AssumeRole` form POST, optionally carrying a second factor.
@@ -15,39 +15,23 @@
//! Regression test for Issue #1423
//! Verifies that Bucket Policies are honored for Authenticated Users.
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::config::{Credentials, Region};
use crate::common::{AdminTransport, RustFSTestEnvironment, admin_create_user_via, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::{Client, Config};
use tracing::info;
/// This suite deliberately drives the admin API through the external `awscurl`
/// binary, so user creation pins `AdminTransport::Awscurl`.
async fn create_user(
env: &RustFSTestEnvironment,
username: &str,
password: &str,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let create_user_body = serde_json::json!({
"secretKey": password,
"status": "enabled"
})
.to_string();
let create_user_url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username);
crate::common::awscurl_put(&create_user_url, &create_user_body, &env.access_key, &env.secret_key).await?;
Ok(())
admin_create_user_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, username, password).await
}
fn create_user_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
let credentials = Credentials::new(access_key, secret_key, None, None, "test-user");
let config = Config::builder()
.credentials_provider(credentials)
.region(Region::new("us-east-1"))
.endpoint_url(&env.url)
.force_path_style(true)
.behavior_version_latest()
.build();
Client::from_conf(config)
env.create_s3_client_with_credentials(access_key, secret_key)
}
#[tokio::test]
+117 -19
View File
@@ -1744,30 +1744,128 @@ pub(crate) async fn admin_create_user(
username: &str,
secret_key: &str,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username);
let body = serde_json::json!({
"secretKey": secret_key,
"status": "enabled"
});
let response = signed_request(
http::Method::PUT,
&url,
&env.access_key,
&env.secret_key,
Some(body.to_string().into_bytes()),
Some("application/json"),
)
.await?;
admin_create_user_via(AdminTransport::Signed, &env.url, &env.access_key, &env.secret_key, username, secret_key).await
}
if response.status() != reqwest::StatusCode::OK {
let status = response.status();
let body = response.text().await.unwrap_or_default();
return Err(format!("create user failed: {status} {body}").into());
/// Transport used by the shared admin-API helpers: in-process SigV4 signing
/// via [`signed_request`], or the external `awscurl` binary (an independent
/// SigV4 implementation exercised by the awscurl-gated suites).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum AdminTransport {
Signed,
Awscurl,
}
/// Execute an admin-API request against `base_url` with admin credentials over
/// the chosen transport, failing on any non-success response.
pub(crate) async fn admin_execute_at(
transport: AdminTransport,
method: http::Method,
base_url: &str,
admin_access_key: &str,
admin_secret_key: &str,
path_and_query: &str,
body: Option<&str>,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let url = format!("{base_url}{path_and_query}");
match transport {
AdminTransport::Signed => {
let content_type = match body {
Some(body) if !body.is_empty() => Some("application/json"),
_ => None,
};
let response = signed_request(
method.clone(),
&url,
admin_access_key,
admin_secret_key,
body.map(|body| body.as_bytes().to_vec()),
content_type,
)
.await?;
if !response.status().is_success() {
let status = response.status();
let text = response.text().await.unwrap_or_default();
return Err(format!("{method} {path_and_query} failed: {status} {text}").into());
}
}
AdminTransport::Awscurl => {
execute_awscurl(&url, method.as_str(), body, admin_access_key, admin_secret_key).await?;
}
}
Ok(())
}
/// Create a new IAM user via the admin API over the chosen transport.
pub(crate) async fn admin_create_user_via(
transport: AdminTransport,
base_url: &str,
admin_access_key: &str,
admin_secret_key: &str,
username: &str,
secret_key: &str,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let path = format!("/rustfs/admin/v3/add-user?accessKey={username}");
let body = serde_json::json!({"secretKey": secret_key, "status": "enabled"}).to_string();
admin_execute_at(
transport,
http::Method::PUT,
base_url,
admin_access_key,
admin_secret_key,
&path,
Some(&body),
)
.await
}
/// Install a canned policy via the admin API over the chosen transport.
pub(crate) async fn admin_add_canned_policy_via(
transport: AdminTransport,
base_url: &str,
admin_access_key: &str,
admin_secret_key: &str,
policy_name: &str,
policy_json: &str,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let path = format!("/rustfs/admin/v3/add-canned-policy?name={policy_name}");
admin_execute_at(
transport,
http::Method::PUT,
base_url,
admin_access_key,
admin_secret_key,
&path,
Some(policy_json),
)
.await
}
/// Attach a canned policy to a user via the admin API over the chosen transport.
pub(crate) async fn admin_attach_user_policy_via(
transport: AdminTransport,
base_url: &str,
admin_access_key: &str,
admin_secret_key: &str,
policy_name: &str,
username: &str,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let path = format!("/rustfs/admin/v3/set-user-or-group-policy?policyName={policy_name}&userOrGroup={username}&isGroup=false");
// `Some("")` preserves the historical wire shape on both transports: awscurl
// keeps sending `-d ''` and the signed path attaches an empty body with no
// content type.
admin_execute_at(
transport,
http::Method::PUT,
base_url,
admin_access_key,
admin_secret_key,
&path,
Some(""),
)
.await
}
#[cfg(test)]
mod tests {
use super::*;
@@ -16,37 +16,29 @@
//! session policy** (`Policy` parameter) via `awscurl --service sts` with explicit
//! `Content-Type: application/x-www-form-urlencoded` on `POST /`.
use crate::common::{RustFSTestEnvironment, awscurl_delete, awscurl_post_sts_form_urlencoded, awscurl_put, init_logging};
use aws_sdk_s3::config::{Credentials, Region};
use crate::common::{
AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via,
awscurl_delete, awscurl_post_sts_form_urlencoded, build_test_s3_config, init_logging,
};
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{Delete, ObjectIdentifier, Tag, Tagging};
use aws_sdk_s3::{Client, Config};
use tracing::info;
use uuid::Uuid;
fn user_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
let credentials = Credentials::new(access_key, secret_key, None, None, "e2e-existing-tag");
let config = Config::builder()
.credentials_provider(credentials)
.region(Region::new("us-east-1"))
.endpoint_url(&env.url)
.force_path_style(true)
.behavior_version_latest()
.build();
Client::from_conf(config)
env.create_s3_client_with_credentials(access_key, secret_key)
}
fn sts_session_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str, session_token: &str) -> Client {
let credentials = Credentials::new(access_key, secret_key, Some(session_token.into()), None, "e2e-sts-session");
let config = Config::builder()
.credentials_provider(credentials)
.region(Region::new("us-east-1"))
.endpoint_url(&env.url)
.force_path_style(true)
.behavior_version_latest()
.build();
Client::from_conf(config)
Client::from_conf(build_test_s3_config(
&env.url,
access_key,
secret_key,
Some(session_token),
"e2e-sts-session",
))
}
fn extract_xml_tag(xml: &str, tag: &str) -> Option<String> {
@@ -77,15 +69,16 @@ async fn assume_role_with_session_policy(
parse_assume_role_credentials(&xml)
}
// This suite deliberately drives the admin API through the external `awscurl`
// binary (an independent SigV4 implementation), so the wrappers below pin
// `AdminTransport::Awscurl`.
async fn admin_create_user(
env: &RustFSTestEnvironment,
username: &str,
password: &str,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let body = serde_json::json!({ "secretKey": password, "status": "enabled" }).to_string();
let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username);
awscurl_put(&url, &body, &env.access_key, &env.secret_key).await?;
Ok(())
admin_create_user_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, username, password).await
}
async fn admin_add_canned_policy(
@@ -93,9 +86,15 @@ async fn admin_add_canned_policy(
policy_name: &str,
policy_json: &str,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", env.url, policy_name);
awscurl_put(&url, policy_json, &env.access_key, &env.secret_key).await?;
Ok(())
admin_add_canned_policy_via(
AdminTransport::Awscurl,
&env.url,
&env.access_key,
&env.secret_key,
policy_name,
policy_json,
)
.await
}
async fn admin_attach_policy_to_user(
@@ -103,12 +102,7 @@ async fn admin_attach_policy_to_user(
policy_name: &str,
username: &str,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let url = format!(
"{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=false",
env.url, policy_name, username
);
awscurl_put(&url, "", &env.access_key, &env.secret_key).await?;
Ok(())
admin_attach_user_policy_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, policy_name, username).await
}
async fn admin_remove_user(env: &RustFSTestEnvironment, username: &str) {
+2 -11
View File
@@ -15,20 +15,11 @@
//! E2E tests for group management (fixes #2028).
use crate::common::{RustFSTestEnvironment, admin_ok, admin_request, init_logging};
use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::{Client, Config};
use aws_sdk_s3::Client;
use tracing::info;
fn create_user_s3_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
let credentials = Credentials::new(access_key, secret_key, None, None, "e2e-group-test");
let config = Config::builder()
.credentials_provider(credentials)
.region(Region::new("us-east-1"))
.endpoint_url(&env.url)
.force_path_style(true)
.behavior_version_latest()
.build();
Client::from_conf(config)
env.create_s3_client_with_credentials(access_key, secret_key)
}
#[tokio::test(flavor = "multi_thread")]
@@ -15,13 +15,13 @@
//! Four-node EC regression gate for inline storage and the inline GET reader.
//!
//! The storage decision is based on shard bytes (256 KiB / 32 KiB objects for
//! the default EC 2+2 geometry), while the GET fast path has its own object-size
//! limits (128 KiB / 16 KiB). A local OTLP/HTTP collector observes the existing
//! reader-path counter without adding a scrape endpoint or production logging.
//! the default EC 2+2 geometry), and the GET fast path follows the persisted
//! inline marker. A local OTLP/HTTP collector observes the existing reader-path
//! counter without adding a scrape endpoint or production logging.
//! One S3 GET can select readers on multiple EC nodes, so the counter tracks
//! distributed reader selection rather than HTTP request count.
use crate::common::{RustFSTestClusterEnvironment, RustFSTestEnvironment, init_logging, local_http_client};
use crate::common::{RustFSTestClusterEnvironment, RustFSTestEnvironment, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
@@ -30,7 +30,7 @@ use aws_sdk_s3::types::{
};
use bytes::Bytes;
use flate2::read::GzDecoder;
use http::header::{CONTENT_ENCODING, HOST};
use http::header::CONTENT_ENCODING;
use http::{Method, Request, Response, StatusCode};
use http_body_util::{BodyExt, Full};
use hyper::body::Incoming;
@@ -42,9 +42,6 @@ use opentelemetry_proto::tonic::metrics::v1::{
Metric, NumberDataPoint, ResourceMetrics, ScopeMetrics, Sum, metric, number_data_point,
};
use prost::Message;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use std::collections::BTreeMap;
use std::convert::Infallible;
use std::error::Error;
@@ -92,6 +89,7 @@ const MPU_PART_1_SIZE: usize = 5 * 1024 * 1024;
const MPU_PART_2_SIZE: usize = 16 * KIB;
const TIER_BUCKET: &str = "inline-fallback-cold-tier";
const TIER_PREFIX: &str = "tiered";
const ALLOW_LOOPBACK_TIER_ENDPOINT_ENV: &str = "RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT";
const MSGPACK_FALLBACK_CONTROL_SERIES: [(&str, &str); 4] = [
(FALLBACK_REQUEST_DIRECTION, "ReadMultipleReq"),
(FALLBACK_RESPONSE_DIRECTION, "ReadMultipleResp"),
@@ -794,12 +792,12 @@ fn metric_attribute(key: &str, value: &str) -> KeyValue {
}
fn boundary_cases(state: VersionState) -> Vec<BoundaryCase> {
let (fast_limit, storage_limit) = match state {
VersionState::Enabled => (16 * KIB, 32 * KIB),
VersionState::Unversioned => (128 * KIB, 256 * KIB),
let storage_limit = match state {
VersionState::Enabled => 32 * KIB,
VersionState::Unversioned => 256 * KIB,
// A suspended bucket stores its null version using the unversioned
// shard threshold, while ObjectInfo keeps version-aware GET semantics.
VersionState::Suspended => (16 * KIB, 256 * KIB),
VersionState::Suspended => 256 * KIB,
};
let mut sizes = vec![0, 16 * KIB - 1, 16 * KIB, 16 * KIB + 1, 32 * KIB - 1, 32 * KIB, 32 * KIB + 1];
if !matches!(state, VersionState::Enabled) {
@@ -820,7 +818,7 @@ fn boundary_cases(state: VersionState) -> Vec<BoundaryCase> {
stored_inline: size <= storage_limit,
expected_reader_path: if size == 0 {
EMPTY
} else if size <= fast_limit {
} else if size <= storage_limit {
INLINE_DIRECT
} else {
LEGACY_DUPLEX
@@ -1262,6 +1260,8 @@ async fn put_two_part_multipart(client: &Client, bucket: &str, key: &str) -> Tes
Ok((body, part2, complete.e_tag().map(str::to_owned)))
}
/// Thin wrapper over [`crate::common::admin_request`], kept local so the call
/// sites below keep their `Option<&str>` body shape.
async fn signed_admin_request(
base_url: &str,
method: Method,
@@ -1270,30 +1270,7 @@ async fn signed_admin_request(
access_key: &str,
secret_key: &str,
) -> TestResult<(reqwest::StatusCode, String)> {
let url = format!("{base_url}{path}");
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
let body_bytes = body.map(|value| value.as_bytes().to_vec()).unwrap_or_default();
let request = http::Request::builder()
.method(method.clone())
.uri(uri)
.header(HOST, authority)
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
let client = local_http_client();
let mut request_builder = client.request(method, url.as_str());
for (name, value) in signed.headers() {
request_builder = request_builder.header(name, value);
}
if !body_bytes.is_empty() {
request_builder = request_builder.body(body_bytes);
}
let response = request_builder.send().await?;
let status = response.status();
let text = response.text().await?;
Ok((status, text))
crate::common::admin_request(base_url, method, path, body.map(str::to_string), access_key, secret_key).await
}
fn unique_tier_name() -> String {
@@ -2124,6 +2101,7 @@ async fn four_node_add_tier_converges() -> TestResult {
cold.create_s3_client().create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
hot.start().await?;
let tier_name = unique_tier_name();
@@ -2142,6 +2120,7 @@ async fn four_node_add_tier_converges_after_offline_node_restart_without_second_
cold.create_s3_client().create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
hot.start().await?;
let tier_name = unique_tier_name();
@@ -2238,6 +2217,7 @@ async fn four_node_manual_transition_distributed_admission_conflict_reports_stat
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
hot.set_env("RUSTFS_SCANNER_ENABLED", "false");
hot.set_env("RUSTFS_SCANNER_CYCLE", "3600");
hot.set_env("RUSTFS_MAX_TRANSITION_WORKERS", "1");
@@ -2380,6 +2360,7 @@ async fn four_node_manual_transition_rollout_non_empty_restart_readback() -> Tes
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
hot.set_env("RUSTFS_SCANNER_ENABLED", "false");
hot.set_env("RUSTFS_SCANNER_CYCLE", "3600");
hot.set_env("RUSTFS_MAX_TRANSITION_WORKERS", "2");
@@ -2484,6 +2465,7 @@ async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls_during_
let collector = OtlpMetricCollector::start().await?;
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
configure_mixed_msgpack_cluster(&mut hot, &collector)?;
hot.set_env("RUSTFS_SCANNER_CYCLE", "1");
hot.set_env("RUSTFS_ILM_PROCESS_TIME", "1");
@@ -2595,6 +2577,7 @@ async fn four_node_transitioned_inline_fallback() -> TestResult {
let collector = OtlpMetricCollector::start().await?;
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
configure_reader_metric_cluster(&mut hot, &collector);
hot.set_env("RUSTFS_SCANNER_CYCLE", "1");
hot.set_env("RUSTFS_ILM_PROCESS_TIME", "1");
@@ -0,0 +1,220 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Anonymous access to SSE-KMS objects under per-key authorization.
//!
//! Locks both halves of the anonymous contract decided in backlog#2028 (D4):
//!
//! - **Enforcement on**: anonymous requests hold no `kms` grants, so a public
//! bucket policy does not let them read SSE-KMS objects or write through an
//! SSE-KMS default-encryption rule. Both fail with `AccessDenied`.
//! - **Enforcement off** (the default): bucket policy alone governs anonymous
//! access, matching the pre-enforcement behavior — public SSE-KMS objects are
//! decrypted and served, and anonymous writes are encrypted under the default
//! key.
//!
//! The denial today is emergent — an empty-account principal falling through to
//! the IAM default deny — so without this file a refactor of principal
//! construction or policy evaluation could silently flip it. Each test carries a
//! plaintext-object positive control: a denial proves nothing while the bucket
//! policy has not propagated.
use super::common::{LocalKMSTestEnvironment, create_key_with_specific_id};
use crate::common::{init_logging, local_http_client};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
};
use std::time::Duration;
type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
const DEFAULT_KEY: &str = "kms-anon-default-key";
const BUCKET: &str = "kms-anon-enforcement";
const PLAIN_OBJECT: &str = "plain.txt";
const ENCRYPTED_OBJECT: &str = "encrypted.txt";
const PAYLOAD: &[u8] = b"kms anonymous enforcement payload";
/// How long a bucket policy change may take to reach the request path.
const POLICY_PROPAGATION: Duration = Duration::from_secs(20);
/// Start a local-KMS server and build the public-bucket fixture.
///
/// The bucket holds a plaintext object (the positive control), an SSE-KMS
/// object, an SSE-KMS default-encryption rule, and a bucket policy opening
/// `GetObject`/`PutObject` to everyone. The enforcement switch defaults to off,
/// so the enforcing case has to set it explicitly.
async fn start_public_sse_kms_bucket(env: &mut LocalKMSTestEnvironment, enforce: bool) -> TestResult {
create_key_with_specific_id(&env.kms_keys_dir, DEFAULT_KEY).await?;
let key_dir = env.kms_keys_dir.clone();
let args = vec![
"--kms-enable",
"--kms-backend",
"local",
"--kms-key-dir",
key_dir.as_str(),
"--kms-default-key-id",
DEFAULT_KEY,
];
let mut envs = vec![("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")];
if enforce {
envs.push(("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "true"));
}
env.base_env.start_rustfs_server_with_env(args, &envs).await?;
env.base_env.create_test_bucket(BUCKET).await?;
let owner = env.base_env.create_s3_client();
owner
.put_object()
.bucket(BUCKET)
.key(PLAIN_OBJECT)
.body(ByteStream::from_static(PAYLOAD))
.send()
.await?;
owner
.put_object()
.bucket(BUCKET)
.key(ENCRYPTED_OBJECT)
.body(ByteStream::from_static(PAYLOAD))
.server_side_encryption(ServerSideEncryption::AwsKms)
.ssekms_key_id(DEFAULT_KEY)
.send()
.await?;
let encryption_config = ServerSideEncryptionConfiguration::builder()
.rules(
ServerSideEncryptionRule::builder()
.apply_server_side_encryption_by_default(
ServerSideEncryptionByDefault::builder()
.sse_algorithm(ServerSideEncryption::AwsKms)
.kms_master_key_id(DEFAULT_KEY)
.build()?,
)
.build(),
)
.build()?;
owner
.put_bucket_encryption()
.bucket(BUCKET)
.server_side_encryption_configuration(encryption_config)
.send()
.await?;
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Sid": "PublicReadWrite",
"Effect": "Allow",
"Principal": "*",
"Action": ["s3:GetObject", "s3:PutObject"],
"Resource": [format!("arn:aws:s3:::{BUCKET}/*")]
}]
})
.to_string();
owner.put_bucket_policy().bucket(BUCKET).policy(&policy).send().await?;
let _ = owner.delete_public_access_block().bucket(BUCKET).send().await;
Ok(())
}
fn object_url(env: &LocalKMSTestEnvironment, key: &str) -> String {
format!("{}/{BUCKET}/{key}", env.base_env.url)
}
async fn anonymous_get(env: &LocalKMSTestEnvironment, key: &str) -> Result<reqwest::Response, reqwest::Error> {
local_http_client().get(object_url(env, key)).send().await
}
async fn anonymous_put(env: &LocalKMSTestEnvironment, key: &str) -> Result<reqwest::Response, reqwest::Error> {
local_http_client().put(object_url(env, key)).body(PAYLOAD).send().await
}
/// Retry the plaintext read until the public bucket policy is live.
async fn wait_for_public_read(env: &LocalKMSTestEnvironment) -> TestResult {
let deadline = tokio::time::Instant::now() + POLICY_PROPAGATION;
loop {
let status = anonymous_get(env, PLAIN_OBJECT).await?.status();
if status.as_u16() == 200 {
return Ok(());
}
if tokio::time::Instant::now() >= deadline {
return Err(format!("positive control never became readable: anonymous GET {PLAIN_OBJECT} -> {status}").into());
}
tokio::time::sleep(Duration::from_millis(500)).await;
}
}
async fn assert_anonymous_denied(response: reqwest::Response, what: &str) -> TestResult {
let status = response.status().as_u16();
let body = response.text().await?;
assert_eq!(status, 403, "{what} must be denied, got {status}: {body}");
assert!(body.contains("AccessDenied"), "{what} must carry AccessDenied: {body}");
Ok(())
}
/// Enforcement on: a public bucket policy does not exempt anonymous requests
/// from per-key authorization, on either the read or the default-encryption
/// write path.
#[tokio::test(flavor = "multi_thread")]
async fn anonymous_sse_kms_denied_under_enforcement() -> TestResult {
init_logging();
let mut env = LocalKMSTestEnvironment::new().await?;
start_public_sse_kms_bucket(&mut env, true).await?;
wait_for_public_read(&env).await?;
let read = anonymous_get(&env, ENCRYPTED_OBJECT).await?;
assert_anonymous_denied(read, "anonymous GET of an SSE-KMS object").await?;
let write = anonymous_put(&env, "anon-write.txt").await?;
assert_anonymous_denied(write, "anonymous PUT through an SSE-KMS default-encryption rule").await?;
Ok(())
}
/// Enforcement off (the default): bucket policy alone governs anonymous access,
/// and the default-encryption rule still encrypts anonymous writes.
#[tokio::test(flavor = "multi_thread")]
async fn anonymous_sse_kms_governed_by_bucket_policy_without_enforcement() -> TestResult {
init_logging();
let mut env = LocalKMSTestEnvironment::new().await?;
start_public_sse_kms_bucket(&mut env, false).await?;
wait_for_public_read(&env).await?;
let read = anonymous_get(&env, ENCRYPTED_OBJECT).await?;
assert_eq!(read.status().as_u16(), 200, "anonymous GET of a public SSE-KMS object must succeed");
assert_eq!(read.bytes().await?.as_ref(), PAYLOAD, "the object must be served decrypted");
let write = anonymous_put(&env, "anon-write.txt").await?;
assert_eq!(write.status().as_u16(), 200, "anonymous PUT to a public bucket must succeed");
let stored = env
.base_env
.create_s3_client()
.head_object()
.bucket(BUCKET)
.key("anon-write.txt")
.send()
.await?;
assert_eq!(
stored.server_side_encryption(),
Some(&ServerSideEncryption::AwsKms),
"the anonymous write must be encrypted by the bucket default rule"
);
Ok(())
}
@@ -66,6 +66,7 @@ const SURVIVOR_KEY: &str = "keep/object.bin";
const TIER_NAME: &str = "KMSCOLD";
const TIER_BUCKET: &str = "kms-ilm-cold-tier";
const TIER_PREFIX: &str = "tiered";
const ALLOW_LOOPBACK_TIER_ENDPOINT_ENV: (&str, &str) = ("RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT", "true");
const TRANSITION_BUCKET: &str = "kms-ilm-transition";
const TRANSITION_KEY: &str = "tier/object.bin";
@@ -80,7 +81,7 @@ const ILM_DEADLINE: StdDuration = StdDuration::from_secs(90);
/// `--kms-default-key-id`, insecure dev defaults). The lifecycle env matches
/// `reliant/lifecycle.rs::fast_lifecycle_env` plus `RUSTFS_ILM_DEBUG_DAY_SECS=2`,
/// so a `Days=1` rule is due about two seconds after the write.
async fn start_enforcing_ilm_server(env: &mut LocalKMSTestEnvironment) -> TestResult {
async fn start_enforcing_ilm_server(env: &mut LocalKMSTestEnvironment, extra_env: &[(&str, &str)]) -> TestResult {
create_key_with_specific_id(&env.kms_keys_dir, SSE_KEY).await?;
let key_dir = env.kms_keys_dir.clone();
@@ -94,13 +95,14 @@ async fn start_enforcing_ilm_server(env: &mut LocalKMSTestEnvironment) -> TestRe
SSE_KEY,
];
let envs = [
let mut envs = vec![
("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true"),
("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "true"),
("RUSTFS_SCANNER_CYCLE", "1"),
("RUSTFS_ILM_PROCESS_TIME", "1"),
("RUSTFS_ILM_DEBUG_DAY_SECS", "2"),
];
envs.extend_from_slice(extra_env);
env.base_env.start_rustfs_server_with_env(args, &envs).await?;
Ok(())
@@ -427,7 +429,7 @@ async fn ilm_expiration_on_sse_kms_bucket_under_enforcement() -> TestResult {
init_logging();
let mut env = LocalKMSTestEnvironment::new().await?;
start_enforcing_ilm_server(&mut env).await?;
start_enforcing_ilm_server(&mut env, &[]).await?;
env.base_env.create_test_bucket(EXPIRY_BUCKET).await?;
let client = env.base_env.create_s3_client();
@@ -499,7 +501,7 @@ async fn ilm_transition_on_sse_kms_bucket_under_enforcement_reads_back() -> Test
// Hot server: Local KMS + enforcement + accelerated lifecycle clock.
let mut env = LocalKMSTestEnvironment::new().await?;
start_enforcing_ilm_server(&mut env).await?;
start_enforcing_ilm_server(&mut env, &[ALLOW_LOOPBACK_TIER_ENDPOINT_ENV]).await?;
let hot_client = env.base_env.create_s3_client();
add_rustfs_tier(&env.base_env, &cold.base_env).await?;
+3
View File
@@ -57,6 +57,9 @@ mod copy_object_version_restore_sse_test;
#[cfg(test)]
mod configured_roundtrip_test;
#[cfg(test)]
mod kms_anonymous_enforcement_test;
#[cfg(test)]
mod kms_authorization_negative_matrix_test;
@@ -38,14 +38,10 @@ use aws_sdk_s3::types::{
NotificationConfiguration, NotificationConfigurationFilter, ObjectIdentifier, QueueConfiguration, S3KeyFilter,
VersioningConfiguration,
};
use http::header::{CONTENT_TYPE, HOST};
use local_ip_address::local_ip;
use reqwest::StatusCode;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use rustfs_utils::egress::ENV_OUTBOUND_ALLOW_ORIGINS;
use rustfs_utils::http::headers::{AMZ_REQUEST_ID, REQUEST_ID_HEADER};
use s3s::Body;
use serde_json::Value;
use std::error::Error;
use std::io::Cursor;
@@ -415,42 +411,16 @@ async fn collect_until(
// Admin target configuration (signed admin HTTP)
// ---------------------------------------------------------------------------
/// Thin wrapper over [`crate::common::signed_request`] with this suite's
/// root credentials; a `Some` body is always JSON here.
async fn signed_admin_request(
env: &RustFSTestEnvironment,
method: http::Method,
url: &str,
body: Option<Vec<u8>>,
) -> Result<reqwest::Response, BoxError> {
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
let mut builder = http::Request::builder()
.method(method.clone())
.uri(uri)
.header(HOST, authority)
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
if body.is_some() {
builder = builder.header(CONTENT_TYPE, "application/json");
}
let content_len = body.as_ref().map(|b| b.len() as i64).unwrap_or_default();
let signed = sign_v4(
builder.body(Body::empty())?,
content_len,
&env.access_key,
&env.secret_key,
"",
"us-east-1",
);
let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
let mut request = crate::common::local_http_client().request(reqwest_method, url);
for (name, value) in signed.headers() {
request = request.header(name, value);
}
if let Some(body) = body {
request = request.body(body);
}
Ok(request.send().await?)
let content_type = body.is_some().then_some("application/json");
crate::common::signed_request(method, url, &env.access_key, &env.secret_key, body, content_type).await
}
async fn enable_notify_module(env: &RustFSTestEnvironment) -> TestResult {
@@ -15,28 +15,24 @@
//! Tests for AWS IAM policy variables with single-value, multi-value, and nested scenarios
use crate::common::{
RustFSTestEnvironment, awscurl_delete, awscurl_put, build_test_s3_config, build_test_sts_client, init_logging,
AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via,
awscurl_delete, awscurl_put, build_test_s3_config, build_test_sts_client, init_logging,
};
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::primitives::ByteStream;
use tracing::info;
/// Helper function to create a regular user with given credentials
/// Helper function to create a regular user with given credentials.
///
/// This suite deliberately drives the admin API through the external `awscurl`
/// binary, so the shared helpers are pinned to `AdminTransport::Awscurl`.
async fn create_user(
env: &RustFSTestEnvironment,
username: &str,
password: &str,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let create_user_body = serde_json::json!({
"secretKey": password,
"status": "enabled"
})
.to_string();
let create_user_url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username);
awscurl_put(&create_user_url, &create_user_body, &env.access_key, &env.secret_key).await?;
Ok(())
admin_create_user_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, username, password).await
}
/// Helper function to create and attach a policy
@@ -46,18 +42,17 @@ async fn create_and_attach_policy(
username: &str,
policy_document: serde_json::Value,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let policy_string = policy_document.to_string();
// Create policy
let add_policy_url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", env.url, policy_name);
awscurl_put(&add_policy_url, &policy_string, &env.access_key, &env.secret_key).await?;
// Attach policy to user
let attach_policy_url = format!(
"{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=false",
env.url, policy_name, username
);
awscurl_put(&attach_policy_url, "", &env.access_key, &env.secret_key).await?;
admin_add_canned_policy_via(
AdminTransport::Awscurl,
&env.url,
&env.access_key,
&env.secret_key,
policy_name,
&policy_document.to_string(),
)
.await?;
admin_attach_user_policy_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, policy_name, username)
.await?;
Ok(())
}
+30 -83
View File
@@ -31,15 +31,11 @@
//!
//! Advisory: <https://github.com/rustfs/rustfs/security/advisories/GHSA-3p3x-734c-h5vx>
use crate::common::local_http_client;
use crate::common::rustfs_binary_path_with_features;
use crate::common::{AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via};
use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ProtocolTestEnvironment};
use anyhow::Result;
use http::header::{CONTENT_TYPE, HOST};
use reqwest::Client;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use tokio::process::Command;
use tracing::info;
@@ -67,92 +63,43 @@ fn basic_auth_header_for(access_key: &str, secret_key: &str) -> String {
format!("Basic {}", encoded)
}
async fn signed_admin_request(
method: http::Method,
url: &str,
body: Option<Vec<u8>>,
content_type: Option<&str>,
) -> Result<reqwest::Response> {
let uri = url.parse::<http::Uri>()?;
let authority = uri
.authority()
.ok_or_else(|| anyhow::anyhow!("request URL missing authority"))?
.to_string();
let mut request = http::Request::builder().method(method.clone()).uri(uri);
request = request.header(HOST, authority);
request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
if let Some(content_type) = content_type {
request = request.header(CONTENT_TYPE, content_type);
}
let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default();
let signed = sign_v4(
request.body(Body::empty())?,
content_len,
async fn admin_create_user(base_url: &str, username: &str, secret_key: &str) -> Result<()> {
admin_create_user_via(
AdminTransport::Signed,
base_url,
DEFAULT_ACCESS_KEY,
DEFAULT_SECRET_KEY,
"",
"us-east-1",
);
let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
let mut request_builder = local_http_client().request(reqwest_method, url);
for (name, value) in signed.headers() {
request_builder = request_builder.header(name, value);
}
if let Some(body) = body {
request_builder = request_builder.body(body);
}
Ok(request_builder.send().await?)
}
async fn admin_create_user(base_url: &str, username: &str, secret_key: &str) -> Result<()> {
let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", base_url, username);
let body = serde_json::json!({
"secretKey": secret_key,
"status": "enabled"
});
let response =
signed_admin_request(http::Method::PUT, &url, Some(body.to_string().into_bytes()), Some("application/json")).await?;
if response.status() != reqwest::StatusCode::OK {
let status = response.status();
let body = response.text().await.unwrap_or_default();
anyhow::bail!("create user failed: {status} {body}");
}
Ok(())
username,
secret_key,
)
.await
.map_err(|e| anyhow::anyhow!(e))
}
async fn admin_add_canned_policy(base_url: &str, policy_name: &str, policy: &serde_json::Value) -> Result<()> {
let url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", base_url, policy_name);
let response =
signed_admin_request(http::Method::PUT, &url, Some(policy.to_string().into_bytes()), Some("application/json")).await?;
if response.status() != reqwest::StatusCode::OK {
let status = response.status();
let body = response.text().await.unwrap_or_default();
anyhow::bail!("add canned policy failed: {status} {body}");
}
Ok(())
admin_add_canned_policy_via(
AdminTransport::Signed,
base_url,
DEFAULT_ACCESS_KEY,
DEFAULT_SECRET_KEY,
policy_name,
&policy.to_string(),
)
.await
.map_err(|e| anyhow::anyhow!(e))
}
async fn admin_attach_policy_to_user(base_url: &str, policy_name: &str, username: &str) -> Result<()> {
let url = format!(
"{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=false",
base_url, policy_name, username
);
let response = signed_admin_request(http::Method::PUT, &url, Some(Vec::new()), None).await?;
if response.status() != reqwest::StatusCode::OK {
let status = response.status();
let body = response.text().await.unwrap_or_default();
anyhow::bail!("attach policy failed: {status} {body}");
}
Ok(())
admin_attach_user_policy_via(
AdminTransport::Signed,
base_url,
DEFAULT_ACCESS_KEY,
DEFAULT_SECRET_KEY,
policy_name,
username,
)
.await
.map_err(|e| anyhow::anyhow!(e))
}
/// Test WebDAV: MKCOL (create bucket), PUT, GET, DELETE, PROPFIND operations
+41 -65
View File
@@ -23,9 +23,9 @@
//!
//! There are no containers, no external S3 backend and no `awscurl`: the
//! `AddTier` admin call is signed in-process with `rustfs_signer`, exactly like
//! the other admin-API e2e suites in this crate. The RustFS warm backend has no
//! loopback/SSRF restriction (that guard is replication-only), so `hot` can tier
//! to `cold` over `http://127.0.0.1:<port>`.
//! the other admin-API e2e suites in this crate. The source server uses the
//! explicit test-only loopback opt-in to tier to `cold` over
//! `http://127.0.0.1:<port>` while production keeps the SSRF guard enabled.
//!
//! The hermetic tests drive the transition and restore paths and pin the
//! chains required by ilm-7 and the restore follow-up:
@@ -46,7 +46,7 @@
//! retry serves the object locally until expiry, and expiry leaves the
//! remote object available for a second restore.
use crate::common::{RustFSTestEnvironment, local_http_client};
use crate::common::RustFSTestEnvironment;
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::primitives::ByteStream;
@@ -56,10 +56,6 @@ use aws_sdk_s3::types::{
VersioningConfiguration,
};
use http::Method;
use http::header::HOST;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serde::Deserialize;
use std::time::{Duration as StdDuration, Instant};
use time::{OffsetDateTime, format_description::well_known::Rfc3339};
@@ -100,6 +96,7 @@ const MANUAL_ACTIVE_CANCEL_OBJECTS: usize = 512;
const MANUAL_RESTART_CANCEL_OBJECTS: usize = 512;
const MANUAL_ACTIVE_CANCEL_RUNNING_TIMEOUT: StdDuration = StdDuration::from_secs(15);
const MANUAL_TRANSITION_CANCEL_BARRIER_ENV: &str = "RUSTFS_E2E_MANUAL_TRANSITION_CANCEL_BARRIER";
const ALLOW_LOOPBACK_TIER_ENDPOINT_ENV: (&str, &str) = ("RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT", "true");
const MANUAL_ASYNC_CONFLICT_TERMINAL_TIMEOUT: StdDuration = StdDuration::from_secs(90);
const MANUAL_RESTART_RECOVERY_TIMEOUT: StdDuration = StdDuration::from_secs(80);
const OBJECT_KEY: &str = "tier/鲁A12345/report.bin";
@@ -116,6 +113,20 @@ const HDR_SOURCE_REPLICATION_REQUEST: &str = "x-rustfs-source-replication-reques
const HDR_SOURCE_MTIME: &str = "x-rustfs-source-mtime";
const TIER_MUTATION_RECOVERY_CHANGED: &str = "Remote tier mutation recovery changed before publish";
async fn start_tier_source(hot: &mut RustFSTestEnvironment, extra_env: &[(&str, &str)]) -> TestResult {
let mut env = Vec::with_capacity(extra_env.len() + 1);
env.push(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV);
env.extend_from_slice(extra_env);
hot.start_rustfs_server_with_env(vec![], &env).await
}
async fn restart_tier_source(hot: &mut RustFSTestEnvironment, extra_env: &[(&str, &str)]) -> TestResult {
let mut env = Vec::with_capacity(extra_env.len() + 1);
env.push(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV);
env.extend_from_slice(extra_env);
hot.restart_server_preserving_data(vec![], &env).await
}
/// 5 MiB — the S3 minimum size for a non-final multipart part; the object's only
/// internal part boundary sits at this offset.
const PART0_SIZE: usize = 5 * 1024 * 1024;
@@ -131,9 +142,8 @@ fn payload() -> Vec<u8> {
/// Sign and send an admin request in-process (no `awscurl`).
///
/// Mirrors the shared admin-API e2e pattern: the SigV4 signature is computed
/// over `UNSIGNED_PAYLOAD`, so the JSON body rides on the wire without being
/// pre-hashed. Returns the response status and body text.
/// Thin wrapper over [`crate::common::admin_request`], kept local so the call
/// sites below keep their `Option<&str>` body shape.
async fn signed_admin_request(
base_url: &str,
method: Method,
@@ -142,30 +152,7 @@ async fn signed_admin_request(
access_key: &str,
secret_key: &str,
) -> Result<(reqwest::StatusCode, String), Box<dyn std::error::Error + Send + Sync>> {
let url = format!("{base_url}{path}");
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
let body_bytes = body.map(|b| b.as_bytes().to_vec()).unwrap_or_default();
let request = http::Request::builder()
.method(method.clone())
.uri(uri)
.header(HOST, authority)
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
let client = local_http_client();
let mut request_builder = client.request(method, url.as_str());
for (name, value) in signed.headers() {
request_builder = request_builder.header(name, value);
}
if !body_bytes.is_empty() {
request_builder = request_builder.body(body_bytes);
}
let response = request_builder.send().await?;
let status = response.status();
let text = response.text().await?;
Ok((status, text))
crate::common::admin_request(base_url, method, path, body.map(str::to_string), access_key, secret_key).await
}
/// Wire `hot` -> `cold` as a `TierType::RustFS` remote tier via `AddTier`.
@@ -888,8 +875,7 @@ async fn test_hermetic_transition_main_path() -> TestResult {
// Hot/source server. A 1s scanner cycle is a backstop; transition is
// primarily driven immediately by the multipart completion path.
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_CYCLE", "1")])
.await?;
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_CYCLE", "1")]).await?;
let hot_client = hot.create_s3_client();
// Wire the RustFS remote tier (real connectivity probe, no force).
@@ -987,8 +973,7 @@ async fn test_hermetic_transition_restore_failure_expiry_and_retry() -> TestResu
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_ILM_DEBUG_DAY_SECS", "5")])
.await?;
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_ILM_DEBUG_DAY_SECS", "5")]).await?;
let hot_client = hot.create_s3_client();
add_rustfs_tier(&hot, &cold).await?;
@@ -1116,8 +1101,7 @@ async fn test_manual_transition_run_black_box_semantics() -> TestResult {
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
.await?;
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
let hot_client = hot.create_s3_client();
add_rustfs_tier(&hot, &cold).await?;
let due_mtime = OffsetDateTime::now_utc() - time::Duration::hours(25);
@@ -1222,8 +1206,7 @@ async fn test_manual_transition_async_job_status_polling() -> TestResult {
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
.await?;
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
let hot_client = hot.create_s3_client();
add_rustfs_tier(&hot, &cold).await?;
@@ -1321,8 +1304,7 @@ async fn test_manual_transition_async_limit_reports_terminal_partial() -> TestRe
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
.await?;
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
let hot_client = hot.create_s3_client();
add_rustfs_tier(&hot, &cold).await?;
@@ -1483,8 +1465,8 @@ async fn test_manual_transition_async_scope_conflicts_report_active_job() -> Tes
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(
vec![],
start_tier_source(
&mut hot,
&[
("RUSTFS_SCANNER_ENABLED", "false"),
("RUSTFS_SCANNER_CYCLE", "3600"),
@@ -1593,8 +1575,7 @@ async fn test_manual_transition_async_different_buckets_admit_concurrently() ->
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
.await?;
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
let hot_client = hot.create_s3_client();
add_rustfs_tier(&hot, &cold).await?;
@@ -1714,8 +1695,7 @@ async fn test_manual_transition_async_tier_failure_reports_terminal_partial() ->
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
.await?;
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
let hot_client = hot.create_s3_client();
add_rustfs_tier(&hot, &cold).await?;
@@ -1807,8 +1787,7 @@ async fn test_manual_transition_async_worker_failure_reports_terminal_partial()
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
.await?;
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
let hot_client = hot.create_s3_client();
add_rustfs_tier(&hot, &cold).await?;
cold.stop_server();
@@ -1900,8 +1879,8 @@ async fn test_manual_transition_async_active_cancel_reports_terminal_cancelled()
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(
vec![],
start_tier_source(
&mut hot,
&[
("RUSTFS_SCANNER_ENABLED", "false"),
("RUSTFS_SCANNER_CYCLE", "3600"),
@@ -2006,7 +1985,7 @@ async fn test_manual_transition_async_cancel_after_process_restart_recovers_term
("RUSTFS_TRANSITION_QUEUE_CAPACITY", "512"),
];
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(vec![], &restart_env).await?;
start_tier_source(&mut hot, &restart_env).await?;
let hot_client = hot.create_s3_client();
add_rustfs_tier(&hot, &cold).await?;
@@ -2034,7 +2013,7 @@ async fn test_manual_transition_async_cancel_after_process_restart_recovers_term
.ok_or("async response must include status_endpoint")?;
assert_eq!(accepted.cancel_endpoint.as_deref(), Some(status_endpoint));
hot.restart_server_preserving_data(vec![], &restart_env).await?;
restart_tier_source(&mut hot, &restart_env).await?;
let restarted = manual_transition_job_status(&hot, status_endpoint).await?;
assert_eq!(restarted.job_id, job_id);
@@ -2158,8 +2137,7 @@ async fn test_manual_transition_run_contract_no_status_cancel_fields() -> TestRe
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
.await?;
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
let hot_client = hot.create_s3_client();
add_rustfs_tier(&hot, &cold).await?;
@@ -2197,8 +2175,7 @@ async fn test_manual_transition_run_continuation_token_resumes_without_raw_marke
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
.await?;
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
let hot_client = hot.create_s3_client();
add_rustfs_tier(&hot, &cold).await?;
@@ -2238,8 +2215,7 @@ async fn test_manual_transition_run_continuation_token_resumes_without_raw_marke
"continuation token must not expose the raw object prefix: {continuation}"
);
hot.restart_server_preserving_data(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
.await?;
restart_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
let second = manual_transition_run_with_max_and_continuation(
&hot,
@@ -2272,8 +2248,8 @@ async fn test_manual_transition_run_queue_pressure_partial() -> TestResult {
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(
vec![],
start_tier_source(
&mut hot,
&[
("RUSTFS_SCANNER_ENABLED", "false"),
("RUSTFS_SCANNER_CYCLE", "3600"),
+133 -41
View File
@@ -13,8 +13,9 @@
// limitations under the License.
use crate::common::{
RustFSTestEnvironment, admin_create_user, awscurl_post_sts_form_urlencoded, init_logging, local_http_client,
replication_fast_env, rustfs_binary_path, signed_request, signed_request_with_client, signed_request_with_session_token,
AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user,
awscurl_post_sts_form_urlencoded, init_logging, local_http_client, replication_fast_env, rustfs_binary_path, signed_request,
signed_request_with_client, signed_request_with_session_token,
};
use crate::fake_s3_target::{
FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction as FakeTargetFault, Operation as FakeTargetOperation,
@@ -25,7 +26,7 @@ use crate::kms::common::{
sse_customer_key_md5_base64,
};
use crate::storage_api::replication_extension::BucketTargetSys;
use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::operation::list_object_versions::ListObjectVersionsOutput;
use aws_sdk_s3::primitives::ByteStream;
@@ -33,7 +34,6 @@ use aws_sdk_s3::types::{
BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, DeleteMarkerEntry, ObjectVersion, ServerSideEncryption,
VersioningConfiguration,
};
use aws_sdk_s3::{Client, Config};
use base64_simd::STANDARD as BASE64_STANDARD;
use bytes::Bytes;
use flate2::read::GzDecoder;
@@ -895,15 +895,7 @@ async fn wait_for_replicated_object_over_https(
}
fn create_user_s3_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
let credentials = Credentials::new(access_key, secret_key, None, None, "e2e-site-replication");
let config = Config::builder()
.credentials_provider(credentials)
.region(Region::new("us-east-1"))
.endpoint_url(&env.url)
.force_path_style(true)
.behavior_version_latest()
.build();
Client::from_conf(config)
env.create_s3_client_with_credentials(access_key, secret_key)
}
async fn admin_add_canned_policy(
@@ -911,24 +903,15 @@ async fn admin_add_canned_policy(
policy_name: &str,
policy: &serde_json::Value,
) -> Result<(), Box<dyn Error + Send + Sync>> {
let url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", env.url, policy_name);
let response = signed_request(
http::Method::PUT,
&url,
admin_add_canned_policy_via(
AdminTransport::Signed,
&env.url,
&env.access_key,
&env.secret_key,
Some(policy.to_string().into_bytes()),
Some("application/json"),
policy_name,
&policy.to_string(),
)
.await?;
if response.status() != StatusCode::OK {
let status = response.status();
let body = response.text().await.unwrap_or_default();
return Err(format!("add canned policy failed: {status} {body}").into());
}
Ok(())
.await
}
async fn admin_attach_policy_to_user(
@@ -936,19 +919,7 @@ async fn admin_attach_policy_to_user(
policy_name: &str,
username: &str,
) -> Result<(), Box<dyn Error + Send + Sync>> {
let url = format!(
"{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=false",
env.url, policy_name, username
);
let response = signed_request(http::Method::PUT, &url, &env.access_key, &env.secret_key, Some(Vec::new()), None).await?;
if response.status() != StatusCode::OK {
let status = response.status();
let body = response.text().await.unwrap_or_default();
return Err(format!("attach policy to user failed: {status} {body}").into());
}
Ok(())
admin_attach_user_policy_via(AdminTransport::Signed, &env.url, &env.access_key, &env.secret_key, policy_name, username).await
}
async fn admin_update_group_members(
@@ -1941,6 +1912,21 @@ async fn site_replication_info(env: &RustFSTestEnvironment) -> Result<SiteReplic
Ok(serde_json::from_slice(&response.bytes().await?)?)
}
async fn site_replication_rotate_svc_acct(
env: &RustFSTestEnvironment,
) -> Result<ReplicateEditStatus, Box<dyn Error + Send + Sync>> {
let url = format!("{}/rustfs/admin/v3/site-replication/rotate-svc-acct", env.url);
let response = signed_request(http::Method::POST, &url, &env.access_key, &env.secret_key, None, None).await?;
if response.status() != StatusCode::OK {
let status = response.status();
let body = response.text().await.unwrap_or_default();
return Err(format!("site replication rotate-svc-acct failed: {status} {body}").into());
}
Ok(serde_json::from_slice(&response.bytes().await?)?)
}
async fn site_replication_resync_op(
env: &RustFSTestEnvironment,
operation: &str,
@@ -6339,6 +6325,112 @@ async fn test_site_replication_remove_all_real_dual_node() -> Result<(), Box<dyn
Ok(())
}
#[tokio::test]
async fn test_site_replication_rotate_svc_acct_completes_and_replication_survives_real_dual_node()
-> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut source_env = RustFSTestEnvironment::new().await?;
source_env
.start_rustfs_server_with_env(vec![], LOOPBACK_REPLICATION_TARGET_ENV)
.await?;
let mut target_env = RustFSTestEnvironment::new().await?;
target_env
.start_rustfs_server_without_cleanup_with_env(LOOPBACK_REPLICATION_TARGET_ENV)
.await?;
let source_client = source_env.create_s3_client();
let target_client = target_env.create_s3_client();
let bucket = "site-repl-rotate-svc-acct";
let add_status = site_replication_add(
&source_env,
&[
PeerSite {
name: "source-site".to_string(),
endpoint: source_env.url.clone(),
access_key: source_env.access_key.clone(),
secret_key: source_env.secret_key.clone(),
..Default::default()
},
PeerSite {
name: "target-site".to_string(),
endpoint: target_env.url.clone(),
access_key: target_env.access_key.clone(),
secret_key: target_env.secret_key.clone(),
..Default::default()
},
],
)
.await?;
assert!(add_status.success, "unexpected site add result: {add_status:?}");
let _source_info = wait_for_site_replication_enabled(&source_env, 2).await?;
let _target_info = wait_for_site_replication_enabled(&target_env, 2).await?;
source_client.create_bucket().bucket(bucket).send().await?;
enable_bucket_versioning(&source_env, bucket).await?;
wait_for_bucket_on_target(&target_client, bucket).await?;
let baseline_payload = b"before rotation".to_vec();
source_client
.put_object()
.bucket(bucket)
.key("before-rotate.txt")
.body(ByteStream::from(baseline_payload.clone()))
.send()
.await?;
let replicated_baseline = wait_for_object_on_target(&target_client, bucket, "before-rotate.txt").await?;
assert_eq!(replicated_baseline, baseline_payload);
// A single rotation call must finish the whole hand-over. Before the fix
// the join push could only sign with the freshly installed secret, every
// peer rejected it, the rotation stayed pending forever, and both
// replication directions were dead until an operator retried.
let rotate_status = site_replication_rotate_svc_acct(&source_env).await?;
assert!(rotate_status.success, "rotation did not complete in one call: {rotate_status:?}");
for env in [&source_env, &target_env] {
let deadline = std::time::Instant::now() + Duration::from_secs(30);
loop {
let info = site_replication_info(env).await?;
if info.enabled && info.pending_operation.is_none() {
break;
}
if std::time::Instant::now() > deadline {
return Err(format!("rotation left {} with a pending operation: {:?}", env.url, info.pending_operation).into());
}
tokio::time::sleep(Duration::from_millis(500)).await;
}
}
// Replication must actually flow again in both directions with the
// rotated service-account secret.
let forward_payload = b"after rotation from source".to_vec();
source_client
.put_object()
.bucket(bucket)
.key("after-rotate-forward.txt")
.body(ByteStream::from(forward_payload.clone()))
.send()
.await?;
let replicated_forward = wait_for_object_on_target(&target_client, bucket, "after-rotate-forward.txt").await?;
assert_eq!(replicated_forward, forward_payload);
let reverse_payload = b"after rotation from target".to_vec();
target_client
.put_object()
.bucket(bucket)
.key("after-rotate-reverse.txt")
.body(ByteStream::from(reverse_payload.clone()))
.send()
.await?;
let replicated_reverse = wait_for_object_on_target(&source_client, bucket, "after-rotate-reverse.txt").await?;
assert_eq!(replicated_reverse, reverse_payload);
Ok(())
}
#[tokio::test]
async fn test_site_replication_state_edit_fresh_and_stale_real_dual_node() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
+35 -1
View File
@@ -352,6 +352,28 @@ pub struct BucketTargetSys {
heartbeat_started: OnceLock<()>,
}
/// Build the bucket-target health-check HTTP client without panicking when
/// the host has no system CA bundle (issue #6734).
///
/// `BucketTargetSys::get()` initializes lazily on the startup path (bucket
/// metadata install calls it on the main thread), and `reqwest::Client::new()`
/// panics when the TLS backend cannot load any system trust root — the state
/// of a minimal container image. Fall back to a client with an explicit empty
/// trust store: HTTP health checks keep working, and HTTPS targets fail closed
/// at the TLS handshake with a clear certificate error instead of aborting
/// the whole process at startup.
fn build_health_check_client() -> HttpClient {
HttpClient::builder().build().unwrap_or_else(|error| {
warn!(
"bucket target health-check HTTP client could not load system TLS roots ({error}); continuing with an empty trust store — HTTPS target health checks will fail until a CA bundle is installed"
);
HttpClient::builder()
.tls_certs_only(std::iter::empty::<reqwest::Certificate>())
.build()
.expect("HTTP client construction must succeed with an explicit empty trust store")
})
}
impl BucketTargetSys {
pub fn get() -> &'static Self {
GLOBAL_BUCKET_TARGET_SYS.get_or_init(Self::new)
@@ -364,7 +386,7 @@ impl BucketTargetSys {
targets_map: Arc::new(RwLock::new(HashMap::new())),
h_mutex: Arc::new(RwLock::new(HashMap::new())),
target_h_mutex: Arc::new(RwLock::new(HashMap::new())),
hc_client: Arc::new(HttpClient::new()),
hc_client: Arc::new(build_health_check_client()),
a_mutex: Arc::new(Mutex::new(HashMap::new())),
arn_errs_map: Arc::new(RwLock::new(HashMap::new())),
target_update_mutexes: Arc::new(Mutex::new(HashMap::new())),
@@ -2490,6 +2512,18 @@ mod tests {
use super::*;
use rcgen::generate_simple_self_signed;
// The startup panic fix for hosts without a CA bundle (issue #6734) rests
// on two properties: the health-check client constructor never panics, and
// its degraded fallback — an explicit empty trust store — always builds.
#[test]
fn health_check_client_construction_never_panics() {
let _ = build_health_check_client();
HttpClient::builder()
.tls_certs_only(std::iter::empty::<reqwest::Certificate>())
.build()
.expect("empty-trust-store client build must succeed without touching system roots");
}
#[derive(Clone, Debug)]
struct RecordingHttpConnector {
request_uris: Arc<std::sync::Mutex<Vec<String>>>,
+1 -2
View File
@@ -20,14 +20,13 @@ mod durable_namespace;
pub mod evaluator;
pub mod manual_transition_job;
mod metadata_boundary;
pub(crate) use metadata_boundary::{LifecycleExpiryConfigs, get_expiry_configs};
pub(crate) use metadata_boundary::{LifecycleExpiryConfigs, get_expiry_configs, get_lifecycle_config};
mod object_handlers_common;
mod object_lock_boundary;
pub use self::core as lifecycle;
mod replication_sink;
pub mod rule;
mod runtime_boundary;
mod tagging_boundary;
pub mod tier_delete_journal;
pub mod tier_free_version_recovery;
pub mod tier_last_day_stats;
@@ -1,37 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::collections::HashMap;
#[allow(
dead_code,
reason = "declared boundary surface for the ECStore replication split plan; no caller in this port (backlog#1823)"
)]
pub(crate) fn decode_tags_to_map(tags: &str) -> HashMap<String, String> {
crate::bucket::tagging::decode_tags_to_map(tags)
}
#[cfg(test)]
mod tests {
use super::decode_tags_to_map;
#[test]
fn decode_tags_to_map_preserves_bucket_tagging_parser_behavior() {
let tags = decode_tags_to_map("env=prod&encoded=a%2Fb&=ignored");
assert_eq!(tags.get("env").map(String::as_str), Some("prod"));
assert_eq!(tags.get("encoded").map(String::as_str), Some("a/b"));
assert!(!tags.contains_key(""));
}
}
+131 -16
View File
@@ -27,7 +27,7 @@ use rustfs_protos::{
ConnectionEvictionLogLevel, evict_failed_connection_with_log_level, models::PingBodyBuilder,
proto_gen::node_service::node_service_client::NodeServiceClient,
};
use std::time::Duration;
use std::{sync::OnceLock, time::Duration};
use tokio::time::timeout;
use tonic::Request;
use tonic::service::interceptor::InterceptedService;
@@ -44,11 +44,35 @@ pub struct RemoteClient {
}
impl RemoteClient {
const ONLINE_CHECK_RESOURCE: &'static str = "health-lock-online";
pub fn new(endpoint: String) -> Self {
Self { addr: endpoint }
}
fn ping_body() -> Bytes {
static BODY: OnceLock<Bytes> = OnceLock::new();
BODY.get_or_init(|| {
let mut fbb = flatbuffers::FlatBufferBuilder::new();
let payload = fbb.create_vector(b"health-check");
let mut builder = PingBodyBuilder::new(&mut fbb);
builder.add_payload(payload);
let root = builder.finish();
fbb.finish(root, None);
Bytes::copy_from_slice(fbb.finished_data())
})
.clone()
}
fn build_ping_request() -> PingRequest {
PingRequest {
version: 1,
body: Self::ping_body(),
}
}
#[cfg(test)]
fn build_fresh_ping_request_for_test() -> PingRequest {
let mut fbb = flatbuffers::FlatBufferBuilder::new();
let payload = fbb.create_vector(b"health-check");
let mut builder = PingBodyBuilder::new(&mut fbb);
@@ -164,6 +188,16 @@ impl RemoteClient {
)
}
fn online_check_timeout() -> Duration {
Duration::from_millis(
rustfs_utils::get_env_u64(
rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS,
rustfs_config::DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS,
)
.max(1),
)
}
async fn execute_rpc<T, F>(&self, op: &'static str, resource_summary: &str, future: F) -> std::result::Result<T, LockError>
where
F: std::future::Future<Output = std::result::Result<T, tonic::Status>>,
@@ -547,24 +581,37 @@ impl LockClient for RemoteClient {
}
async fn is_online(&self) -> bool {
// Use Ping interface to test if remote service is online
let mut client = match self.get_client().await {
Ok(client) => client,
Err(_) => {
info!("remote client {} connection failed", self.addr);
return false;
}
};
let ping_req = Request::new(Self::build_ping_request());
match client.ping(ping_req).await {
Ok(_) => {
info!("remote client {} is online", self.addr);
let online_timeout = Self::online_check_timeout();
match timeout(online_timeout, async {
let mut client = self.get_client().await?;
let ping_req = Request::new(Self::build_ping_request());
self.execute_rpc("ping", Self::ONLINE_CHECK_RESOURCE, client.ping(ping_req))
.await?;
Ok::<(), LockError>(())
})
.await
{
Ok(Ok(())) => {
debug!(addr = %self.addr, timeout_ms = online_timeout.as_millis(), "remote lock client is online");
true
}
Ok(Err(err)) => {
debug!(
addr = %self.addr,
timeout_ms = online_timeout.as_millis(),
error = %err,
"remote lock client online check failed"
);
false
}
Err(_) => {
info!("remote client {} ping failed", self.addr);
let reason = format!("online check timed out after {:?}", online_timeout);
warn!(
addr = %self.addr,
timeout_ms = online_timeout.as_millis(),
"remote lock client online check timed out"
);
self.evict_connection("ping", &reason, Self::ONLINE_CHECK_RESOURCE).await;
false
}
}
@@ -651,6 +698,15 @@ mod tests {
);
}
#[test]
fn cached_ping_request_matches_fresh_flatbuffer_payload() {
let cached = RemoteClient::build_ping_request();
let fresh = RemoteClient::build_fresh_ping_request_for_test();
assert_eq!(cached.version, fresh.version);
assert_eq!(cached.body, fresh.body);
}
#[tokio::test]
#[serial_test::serial]
async fn test_remote_client_acquire_lock_uses_rpc_timeout_and_evicts_connection() {
@@ -779,6 +835,48 @@ mod tests {
accept_task.abort();
}
#[tokio::test]
#[serial_test::serial]
async fn test_remote_client_is_online_uses_health_timeout_and_evicts_connection() {
ensure_test_rpc_secret();
let Some((addr, accept_task)) = spawn_hanging_listener().await else {
return;
};
cache_lazy_channel(&addr).await;
assert!(runtime_sources::test_node_channel_is_cached(&addr).await);
temp_env::async_with_vars(
[
(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("50")),
(rustfs_config::ENV_OBJECT_LOCK_RPC_TIMEOUT_MS, Some("1000")),
],
async {
let client = RemoteClient::new(addr.clone());
let started_at = tokio::time::Instant::now();
let online = client.is_online().await;
let elapsed = started_at.elapsed();
assert!(!online, "hanging remote lock peer must not be reported online");
assert!(
elapsed >= Duration::from_millis(40),
"remote online check should honor configured health timeout, got {elapsed:?}"
);
assert!(
elapsed < Duration::from_secs(1),
"health timeout should keep readiness probes bounded, got {elapsed:?}"
);
assert!(
!runtime_sources::test_node_channel_is_cached(&addr).await,
"online-check timeout should evict cached connection"
);
},
)
.await;
accept_task.abort();
}
#[tokio::test]
#[serial_test::serial]
async fn test_remote_client_refresh_tonic_error_evicts_connection() {
@@ -906,4 +1004,21 @@ mod tests {
assert_eq!(RemoteClient::rpc_timeout(), Duration::from_millis(1));
});
}
#[test]
#[serial_test::serial]
fn test_remote_client_online_timeout_honors_configured_deadline() {
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, None::<&str>, || {
assert_eq!(
RemoteClient::online_check_timeout(),
Duration::from_millis(rustfs_config::DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS)
);
});
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("50"), || {
assert_eq!(RemoteClient::online_check_timeout(), Duration::from_millis(50));
});
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("0"), || {
assert_eq!(RemoteClient::online_check_timeout(), Duration::from_millis(1));
});
}
}
+33 -555
View File
@@ -12,35 +12,28 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use super::target_defaults::{amqp_kvs, kafka_kvs, mysql_kvs, nats_kvs, postgres_kvs, pulsar_kvs, redis_kvs};
use rustfs_config::audit::AUDIT_REDIS_DEFAULT_CHANNEL;
use rustfs_config::server_config::{KV, KVS};
use rustfs_config::{
AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_SASL_ENABLE,
KAFKA_SASL_MECHANISM, KAFKA_SASL_PASSWORD, KAFKA_SASL_USERNAME, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY,
KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR,
MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY,
MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT,
MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT,
MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS,
NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR,
NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN,
NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE,
POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER,
PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT,
REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD,
REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT,
REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL,
REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY,
WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL,
WEBHOOK_SKIP_TLS_VERIFY,
COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY, EVENT_DEFAULT_DIR, EnableState, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD,
MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY,
MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, WEBHOOK_AUTH_TOKEN,
WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT,
WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, WEBHOOK_SKIP_TLS_VERIFY,
};
use std::sync::LazyLock;
#[allow(clippy::declare_interior_mutable_const)]
/// Default KVS for audit webhook settings.
///
/// `WEBHOOK_BATCH_SIZE`/`WEBHOOK_MAX_RETRY`/`WEBHOOK_RETRY_INTERVAL`/`WEBHOOK_HTTP_TIMEOUT`
/// exist here but not in [`crate::config::notify::DEFAULT_NOTIFY_WEBHOOK_KVS`]. This mirrors
/// MinIO upstream: `internal/logger/config.go`'s `DefaultAuditWebhookKVS` carries the same
/// four keys with the same defaults (`"1"`/`"0"`/`"3s"`/`"5s"`), while
/// `internal/config/notify/parse.go`'s `DefaultWebhookKVS` (bucket event notifications) does
/// not — the notify webhook delivery path never supported them. Not a copy/paste gap
/// (backlog#2054).
pub static DEFAULT_AUDIT_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
@@ -56,7 +49,7 @@ pub static DEFAULT_AUDIT_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
KV {
key: WEBHOOK_AUTH_TOKEN.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
hidden_if_empty: true, // Sensitive field; matches notify's webhook auth_token (backlog#2054)
},
KV {
key: WEBHOOK_CLIENT_CERT.to_owned(),
@@ -118,6 +111,15 @@ pub static DEFAULT_AUDIT_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
#[allow(clippy::declare_interior_mutable_const)]
/// Default KVS for audit MQTT settings.
///
/// `MQTT_QOS`/`MQTT_KEEP_ALIVE_INTERVAL`/`MQTT_RECONNECT_INTERVAL` default to a stronger
/// delivery posture here (`"1"`/`"60s"`/`"5s"`) than
/// [`crate::config::notify::DEFAULT_NOTIFY_MQTT_KVS`] (`"0"`/`"0s"`/`"0s"`, which matches
/// MinIO's own `DefaultMQTTKVS` in `internal/config/notify/parse.go` byte-for-byte). MinIO has
/// no MQTT audit target to compare against — audit-over-MQTT is a RustFS-original addition —
/// so this divergence cannot be checked against upstream; it is intentional (audit favors
/// at-least-once delivery and faster reconnect over notify's opt-in defaults), not a
/// copy/paste gap (backlog#2054).
pub static DEFAULT_AUDIT_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
@@ -208,542 +210,18 @@ pub static DEFAULT_AUDIT_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
])
});
pub static DEFAULT_AUDIT_AMQP_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
key: ENABLE_KEY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: AMQP_URL.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: AMQP_EXCHANGE.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: AMQP_ROUTING_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: AMQP_MANDATORY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: AMQP_PERSISTENT.to_owned(),
value: EnableState::On.to_string(),
hidden_if_empty: false,
},
KV {
key: AMQP_USERNAME.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: AMQP_PASSWORD.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: AMQP_TLS_CA.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: AMQP_TLS_CLIENT_CERT.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: AMQP_TLS_CLIENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: AMQP_QUEUE_DIR.to_owned(),
value: EVENT_DEFAULT_DIR.to_owned(),
hidden_if_empty: false,
},
KV {
key: AMQP_QUEUE_LIMIT.to_owned(),
value: DEFAULT_LIMIT.to_string(),
hidden_if_empty: false,
},
KV {
key: COMMENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
])
});
// The remaining targets declare the same defaults as notify, so both sides build them from
// `target_defaults`. Redis and mysql pass in the single default that audit and notify disagree on.
pub static DEFAULT_AUDIT_AMQP_KVS: LazyLock<KVS> = LazyLock::new(amqp_kvs);
pub static DEFAULT_AUDIT_NATS_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
key: ENABLE_KEY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: NATS_ADDRESS.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: NATS_SUBJECT.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: NATS_USERNAME.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: NATS_PASSWORD.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: NATS_TOKEN.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: NATS_CREDENTIALS_FILE.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: NATS_TLS_CA.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: NATS_TLS_CLIENT_CERT.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: NATS_TLS_CLIENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: NATS_TLS_REQUIRED.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: NATS_QUEUE_DIR.to_owned(),
value: EVENT_DEFAULT_DIR.to_owned(),
hidden_if_empty: false,
},
KV {
key: NATS_QUEUE_LIMIT.to_owned(),
value: DEFAULT_LIMIT.to_string(),
hidden_if_empty: false,
},
KV {
key: NATS_JETSTREAM_ENABLE.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: NATS_JETSTREAM_STREAM_NAME.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_owned(),
value: NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(),
hidden_if_empty: false,
},
KV {
key: COMMENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
])
});
pub static DEFAULT_AUDIT_NATS_KVS: LazyLock<KVS> = LazyLock::new(nats_kvs);
pub static DEFAULT_AUDIT_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
key: ENABLE_KEY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: PULSAR_BROKER.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: PULSAR_TOPIC.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: PULSAR_AUTH_TOKEN.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: PULSAR_USERNAME.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: PULSAR_PASSWORD.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: PULSAR_TLS_CA.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: PULSAR_TLS_ALLOW_INSECURE.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: PULSAR_TLS_HOSTNAME_VERIFICATION.to_owned(),
value: EnableState::On.to_string(),
hidden_if_empty: false,
},
KV {
key: PULSAR_QUEUE_DIR.to_owned(),
value: EVENT_DEFAULT_DIR.to_owned(),
hidden_if_empty: false,
},
KV {
key: PULSAR_QUEUE_LIMIT.to_owned(),
value: DEFAULT_LIMIT.to_string(),
hidden_if_empty: false,
},
KV {
key: COMMENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
])
});
pub static DEFAULT_AUDIT_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(pulsar_kvs);
pub static DEFAULT_AUDIT_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
key: ENABLE_KEY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: REDIS_URL.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_CHANNEL.to_owned(),
value: AUDIT_REDIS_DEFAULT_CHANNEL.to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_USERNAME.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_PASSWORD.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: REDIS_KEEP_ALIVE_INTERVAL.to_owned(),
value: "15".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_QUEUE_DIR.to_owned(),
value: EVENT_DEFAULT_DIR.to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_QUEUE_LIMIT.to_owned(),
value: DEFAULT_LIMIT.to_string(),
hidden_if_empty: false,
},
KV {
key: REDIS_MAX_RETRY_ATTEMPTS.to_owned(),
value: "3".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_RECONNECT_RETRY_ATTEMPTS.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_MIN_RETRY_DELAY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_MAX_RETRY_DELAY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_CONNECTION_TIMEOUT.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_RESPONSE_TIMEOUT.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_PIPELINE_BUFFER_SIZE.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_TLS_POLICY.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: REDIS_TLS_CA.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: REDIS_TLS_CLIENT_CERT.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: REDIS_TLS_CLIENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: REDIS_TLS_ALLOW_INSECURE.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: COMMENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
])
});
pub static DEFAULT_AUDIT_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| redis_kvs(AUDIT_REDIS_DEFAULT_CHANNEL));
pub static DEFAULT_AUDIT_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
key: ENABLE_KEY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: POSTGRES_DSN_STRING.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: POSTGRES_TABLE.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: POSTGRES_FORMAT.to_owned(),
value: "namespace".to_owned(),
hidden_if_empty: false,
},
KV {
key: POSTGRES_TLS_REQUIRED.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: POSTGRES_TLS_CA.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: POSTGRES_TLS_CLIENT_CERT.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: POSTGRES_TLS_CLIENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: POSTGRES_QUEUE_DIR.to_owned(),
value: EVENT_DEFAULT_DIR.to_owned(),
hidden_if_empty: false,
},
KV {
key: POSTGRES_QUEUE_LIMIT.to_owned(),
value: DEFAULT_LIMIT.to_string(),
hidden_if_empty: false,
},
KV {
key: COMMENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
])
});
pub static DEFAULT_AUDIT_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(postgres_kvs);
pub static DEFAULT_AUDIT_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
key: ENABLE_KEY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: KAFKA_BROKERS.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: KAFKA_TOPIC.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: KAFKA_ACKS.to_owned(),
value: "1".to_owned(),
hidden_if_empty: false,
},
KV {
key: KAFKA_TLS_ENABLE.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: KAFKA_TLS_CA.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: KAFKA_TLS_CLIENT_CERT.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: KAFKA_TLS_CLIENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: KAFKA_SASL_ENABLE.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: KAFKA_SASL_MECHANISM.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: KAFKA_SASL_USERNAME.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: KAFKA_SASL_PASSWORD.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: KAFKA_QUEUE_DIR.to_owned(),
value: EVENT_DEFAULT_DIR.to_owned(),
hidden_if_empty: false,
},
KV {
key: KAFKA_QUEUE_LIMIT.to_owned(),
value: DEFAULT_LIMIT.to_string(),
hidden_if_empty: false,
},
KV {
key: COMMENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
])
});
pub static DEFAULT_AUDIT_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(kafka_kvs);
pub static DEFAULT_AUDIT_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
key: ENABLE_KEY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: MYSQL_DSN_STRING.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: MYSQL_TABLE.to_owned(),
value: "rustfs_audit_logs".to_owned(),
hidden_if_empty: false,
},
KV {
key: MYSQL_FORMAT.to_owned(),
value: "access".to_owned(),
hidden_if_empty: false,
},
KV {
key: MYSQL_TLS_CA.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: MYSQL_TLS_CLIENT_CERT.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: MYSQL_TLS_CLIENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: MYSQL_QUEUE_DIR.to_owned(),
value: EVENT_DEFAULT_DIR.to_owned(),
hidden_if_empty: false,
},
KV {
key: MYSQL_QUEUE_LIMIT.to_owned(),
value: DEFAULT_LIMIT.to_string(),
hidden_if_empty: false,
},
KV {
key: MYSQL_MAX_OPEN_CONNECTIONS.to_owned(),
value: "2".to_owned(),
hidden_if_empty: false,
},
KV {
key: COMMENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
])
});
pub static DEFAULT_AUDIT_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| mysql_kvs("rustfs_audit_logs"));
+60 -12
View File
@@ -2793,12 +2793,13 @@ where
#[cfg(test)]
mod tests {
use super::{
SERVER_CONFIG_LOCK, ServerConfigSnapshot, apply_dynamic_config_for_sub_sys_with, config_task_join_error,
configs_semantically_equal, decode_server_config_blob, encode_server_config_blob, is_standard_object_server_config,
lookup_configs, new_and_save_server_config, read_config, read_config_no_lock_preserve_empty_with_metadata,
read_config_preserve_empty, read_config_with_metadata, read_config_without_migrate, read_server_config_snapshot,
save_server_config, save_server_config_snapshot, save_server_config_snapshot_with_generation,
server_config_transaction_lock_path, should_warn_ignored_scalar_section, storage_class_kvs_mut,
SERVER_CONFIG_LOCK, ServerConfigSnapshot, apply_dynamic_config_for_sub_sys_with, build_scalar_config_object,
config_task_join_error, configs_semantically_equal, decode_server_config_blob, encode_server_config_blob,
heal_config_descriptor, is_standard_object_server_config, lookup_configs, new_and_save_server_config, read_config,
read_config_no_lock_preserve_empty_with_metadata, read_config_preserve_empty, read_config_with_metadata,
read_config_without_migrate, read_server_config_snapshot, save_server_config, save_server_config_snapshot,
save_server_config_snapshot_with_generation, server_config_transaction_lock_path, should_warn_ignored_scalar_section,
storage_class_kvs_mut,
};
use crate::config::{audit, heal, notify, oidc, scanner};
use crate::disk::endpoint::Endpoint;
@@ -3541,6 +3542,31 @@ mod tests {
cfg
}
/// `Config::new()` (and every decode built on it) reads the process-global
/// `rustfs_config::server_config::DEFAULT_KVS` OnceLock at call time, and
/// other tests in this binary register it via `crate::config::init()`
/// mid-run. Equality assertions must therefore normalize both sides with
/// one snapshot taken after both configs exist, never against a later
/// `Config::new()`.
fn default_kvs_snapshot() -> Option<&'static std::collections::HashMap<String, KVS>> {
rustfs_config::server_config::DEFAULT_KVS.get()
}
/// Fills the default sections `cfg` is missing from an explicit
/// [`default_kvs_snapshot`], mirroring `Config::set_defaults`.
fn filled_with_default_kvs(mut cfg: Config, snapshot: Option<&std::collections::HashMap<String, KVS>>) -> Config {
if let Some(defaults) = snapshot {
for (sub_sys, kvs) in defaults {
cfg.0
.entry(sub_sys.clone())
.or_default()
.entry(DEFAULT_DELIMITER.to_string())
.or_insert_with(|| kvs.clone());
}
}
cfg
}
#[test]
fn test_external_scanner_config_decodes_with_defaults() {
let cfg =
@@ -3630,7 +3656,9 @@ mod tests {
}"#;
let cfg = decode_server_config_blob(seed).expect("root heal null should mean no persisted override");
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
// The heal section may hold registered defaults, so assert on the
// semantic diff instead of the section's presence.
assert!(build_scalar_config_object(&cfg, heal_config_descriptor()).is_empty());
assert!(!is_standard_object_server_config(seed));
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("legacy seed should canonicalize on an authorized save");
@@ -3665,7 +3693,12 @@ mod tests {
let input = format!(r#"{{"version":"33","storageclass":{{"standard":"","rrs":""}},{section}}}"#);
let cfg = decode_server_config_blob(input.as_bytes())
.unwrap_or_else(|err| panic!("legacy scalar section {section} should be ignored, got: {err}"));
assert_eq!(cfg, base, "ignored section {section} must contribute no overrides");
let snapshot = default_kvs_snapshot();
assert_eq!(
filled_with_default_kvs(cfg.clone(), snapshot),
filled_with_default_kvs(base.clone(), snapshot),
"ignored section {section} must contribute no overrides"
);
assert!(
!is_standard_object_server_config(input.as_bytes()),
"seed with {section} must not count as standard so a save rewrites it"
@@ -3743,12 +3776,19 @@ mod tests {
fn valid_heal_object_and_kvs_array_shapes_remain_accepted() {
let empty_object = br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":{}}"#;
let cfg = decode_server_config_blob(empty_object).expect("empty heal object should decode as no override");
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
// The heal section may hold registered defaults, so assert on the
// semantic diff instead of the section's presence.
assert!(build_scalar_config_object(&cfg, heal_config_descriptor()).is_empty());
let kvs_array =
br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":[{"key":"bitrot_cycle","value":"off"}]}"#;
let cfg = decode_server_config_blob(kvs_array).expect("heal KVS array should decode");
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_some());
assert_eq!(
build_scalar_config_object(&cfg, heal_config_descriptor())
.get(HEAL_BITROT_CYCLE)
.and_then(Value::as_str),
Some("off")
);
}
#[test]
@@ -4900,8 +4940,12 @@ mod tests {
fn test_fallback_returns_default_config_when_recovery_enabled() {
let cfg = fallback_server_config_after_corruption(corrupt_config_error(), "config/config.json", true)
.expect("recovery enabled must fall back to the default config");
let snapshot = default_kvs_snapshot();
assert!(
configs_semantically_equal(&cfg, &Config::new()),
configs_semantically_equal(
&filled_with_default_kvs(cfg, snapshot),
&filled_with_default_kvs(Config(std::collections::HashMap::new()), snapshot)
),
"fallback config should be the default server config"
);
}
@@ -5518,8 +5562,12 @@ mod tests {
.expect("unrecoverable corruption should fall back to the default config");
assert_eq!(store.heal_calls.load(Ordering::SeqCst), 1, "heal should be attempted before falling back");
let snapshot = default_kvs_snapshot();
assert!(
configs_semantically_equal(&cfg, &Config::new()),
configs_semantically_equal(
&filled_with_default_kvs(cfg, snapshot),
&filled_with_default_kvs(Config(std::collections::HashMap::new()), snapshot)
),
"fallback config should be the default server config"
);
}
+1
View File
@@ -21,6 +21,7 @@ mod notify;
mod oidc;
mod scanner;
pub mod storageclass;
mod target_defaults;
use crate::error::Result;
use crate::store::ECStore;
+25 -553
View File
@@ -12,34 +12,26 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use super::target_defaults::{amqp_kvs, kafka_kvs, mysql_kvs, nats_kvs, postgres_kvs, pulsar_kvs, redis_kvs};
use rustfs_config::notify::NOTIFY_REDIS_DEFAULT_CHANNEL;
use rustfs_config::server_config::{KV, KVS};
use rustfs_config::{
AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_SASL_ENABLE,
KAFKA_SASL_MECHANISM, KAFKA_SASL_PASSWORD, KAFKA_SASL_USERNAME, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY,
KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR,
MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY,
MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT,
MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT,
MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS,
NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR,
NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN,
NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE,
POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER,
PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT,
REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD,
REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT,
REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL,
REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT,
WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY,
COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY, EVENT_DEFAULT_DIR, EnableState, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD,
MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY,
MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, WEBHOOK_AUTH_TOKEN,
WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT,
WEBHOOK_SKIP_TLS_VERIFY,
};
use std::sync::LazyLock;
/// The default configuration collection of webhooks
/// Initialized only once during the program life cycle, enabling high-performance lazy loading.
///
/// This table has no `batch_size`/`max_retry`/`retry_interval`/`http_timeout` keys, unlike
/// [`crate::config::audit::DEFAULT_AUDIT_WEBHOOK_KVS`] — matching MinIO upstream, whose
/// `internal/config/notify/parse.go` `DefaultWebhookKVS` (bucket event notifications) also
/// omits them while `internal/logger/config.go`'s `DefaultAuditWebhookKVS` carries them.
/// Intentional, not a copy/paste gap (backlog#2054).
pub static DEFAULT_NOTIFY_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
@@ -97,6 +89,12 @@ pub static DEFAULT_NOTIFY_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
});
/// MQTT's default configuration collection
///
/// `MQTT_QOS`/`MQTT_KEEP_ALIVE_INTERVAL`/`MQTT_RECONNECT_INTERVAL` default to `"0"`/`"0s"`/`"0s"`
/// here, matching MinIO's `DefaultMQTTKVS` in `internal/config/notify/parse.go`
/// byte-for-byte — this table is a faithful port. [`crate::config::audit::DEFAULT_AUDIT_MQTT_KVS`]
/// uses stronger, RustFS-original defaults instead (MinIO has no MQTT audit target to compare
/// against); that divergence is intentional, not a copy/paste gap (backlog#2054).
pub static DEFAULT_NOTIFY_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
@@ -188,543 +186,17 @@ pub static DEFAULT_NOTIFY_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
])
});
pub static DEFAULT_NOTIFY_AMQP_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
key: ENABLE_KEY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: AMQP_URL.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: AMQP_EXCHANGE.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: AMQP_ROUTING_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: AMQP_MANDATORY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: AMQP_PERSISTENT.to_owned(),
value: EnableState::On.to_string(),
hidden_if_empty: false,
},
KV {
key: AMQP_USERNAME.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: AMQP_PASSWORD.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: AMQP_TLS_CA.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: AMQP_TLS_CLIENT_CERT.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: AMQP_TLS_CLIENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: AMQP_QUEUE_DIR.to_owned(),
value: EVENT_DEFAULT_DIR.to_owned(),
hidden_if_empty: false,
},
KV {
key: AMQP_QUEUE_LIMIT.to_owned(),
value: DEFAULT_LIMIT.to_string(),
hidden_if_empty: false,
},
KV {
key: COMMENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
])
});
pub static DEFAULT_NOTIFY_AMQP_KVS: LazyLock<KVS> = LazyLock::new(amqp_kvs);
pub static DEFAULT_NOTIFY_NATS_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
key: ENABLE_KEY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: NATS_ADDRESS.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: NATS_SUBJECT.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: NATS_USERNAME.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: NATS_PASSWORD.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: NATS_TOKEN.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: NATS_CREDENTIALS_FILE.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: NATS_TLS_CA.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: NATS_TLS_CLIENT_CERT.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: NATS_TLS_CLIENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: NATS_TLS_REQUIRED.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: NATS_QUEUE_DIR.to_owned(),
value: EVENT_DEFAULT_DIR.to_owned(),
hidden_if_empty: false,
},
KV {
key: NATS_QUEUE_LIMIT.to_owned(),
value: DEFAULT_LIMIT.to_string(),
hidden_if_empty: false,
},
KV {
key: NATS_JETSTREAM_ENABLE.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: NATS_JETSTREAM_STREAM_NAME.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_owned(),
value: NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(),
hidden_if_empty: false,
},
KV {
key: COMMENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
])
});
pub static DEFAULT_NOTIFY_NATS_KVS: LazyLock<KVS> = LazyLock::new(nats_kvs);
pub static DEFAULT_NOTIFY_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
key: ENABLE_KEY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: PULSAR_BROKER.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: PULSAR_TOPIC.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: PULSAR_AUTH_TOKEN.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: PULSAR_USERNAME.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: PULSAR_PASSWORD.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: PULSAR_TLS_CA.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: PULSAR_TLS_ALLOW_INSECURE.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: PULSAR_TLS_HOSTNAME_VERIFICATION.to_owned(),
value: EnableState::On.to_string(),
hidden_if_empty: false,
},
KV {
key: PULSAR_QUEUE_DIR.to_owned(),
value: EVENT_DEFAULT_DIR.to_owned(),
hidden_if_empty: false,
},
KV {
key: PULSAR_QUEUE_LIMIT.to_owned(),
value: DEFAULT_LIMIT.to_string(),
hidden_if_empty: false,
},
KV {
key: COMMENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
])
});
pub static DEFAULT_NOTIFY_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(pulsar_kvs);
pub static DEFAULT_NOTIFY_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
key: ENABLE_KEY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: REDIS_URL.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_CHANNEL.to_owned(),
value: NOTIFY_REDIS_DEFAULT_CHANNEL.to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_USERNAME.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_PASSWORD.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: REDIS_KEEP_ALIVE_INTERVAL.to_owned(),
value: "15".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_QUEUE_DIR.to_owned(),
value: EVENT_DEFAULT_DIR.to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_QUEUE_LIMIT.to_owned(),
value: DEFAULT_LIMIT.to_string(),
hidden_if_empty: false,
},
KV {
key: REDIS_MAX_RETRY_ATTEMPTS.to_owned(),
value: "3".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_RECONNECT_RETRY_ATTEMPTS.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_MIN_RETRY_DELAY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_MAX_RETRY_DELAY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_CONNECTION_TIMEOUT.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_RESPONSE_TIMEOUT.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_PIPELINE_BUFFER_SIZE.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: REDIS_TLS_POLICY.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: REDIS_TLS_CA.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: REDIS_TLS_CLIENT_CERT.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: REDIS_TLS_CLIENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: REDIS_TLS_ALLOW_INSECURE.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: COMMENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
])
});
pub static DEFAULT_NOTIFY_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| redis_kvs(NOTIFY_REDIS_DEFAULT_CHANNEL));
pub static DEFAULT_NOTIFY_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
key: ENABLE_KEY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: POSTGRES_DSN_STRING.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: POSTGRES_TABLE.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: POSTGRES_FORMAT.to_owned(),
value: "namespace".to_owned(),
hidden_if_empty: false,
},
KV {
key: POSTGRES_TLS_REQUIRED.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: POSTGRES_TLS_CA.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: POSTGRES_TLS_CLIENT_CERT.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: POSTGRES_TLS_CLIENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: POSTGRES_QUEUE_DIR.to_owned(),
value: EVENT_DEFAULT_DIR.to_owned(),
hidden_if_empty: false,
},
KV {
key: POSTGRES_QUEUE_LIMIT.to_owned(),
value: DEFAULT_LIMIT.to_string(),
hidden_if_empty: false,
},
KV {
key: COMMENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
])
});
pub static DEFAULT_NOTIFY_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(postgres_kvs);
pub static DEFAULT_NOTIFY_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
key: ENABLE_KEY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: KAFKA_BROKERS.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: KAFKA_TOPIC.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: KAFKA_ACKS.to_owned(),
value: "1".to_owned(),
hidden_if_empty: false,
},
KV {
key: KAFKA_TLS_ENABLE.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: KAFKA_TLS_CA.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: KAFKA_TLS_CLIENT_CERT.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: KAFKA_TLS_CLIENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: KAFKA_SASL_ENABLE.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: KAFKA_SASL_MECHANISM.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: KAFKA_SASL_USERNAME.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
KV {
key: KAFKA_SASL_PASSWORD.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: KAFKA_QUEUE_DIR.to_owned(),
value: EVENT_DEFAULT_DIR.to_owned(),
hidden_if_empty: false,
},
KV {
key: KAFKA_QUEUE_LIMIT.to_owned(),
value: DEFAULT_LIMIT.to_string(),
hidden_if_empty: false,
},
KV {
key: COMMENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
])
});
pub static DEFAULT_NOTIFY_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(kafka_kvs);
/// MySQL notification target default configuration
pub static DEFAULT_NOTIFY_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
key: ENABLE_KEY.to_owned(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
KV {
key: MYSQL_DSN_STRING.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: MYSQL_TABLE.to_owned(),
value: "rustfs_events".to_owned(),
hidden_if_empty: false,
},
KV {
key: MYSQL_FORMAT.to_owned(),
value: "access".to_owned(),
hidden_if_empty: false,
},
KV {
key: MYSQL_TLS_CA.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: MYSQL_TLS_CLIENT_CERT.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: MYSQL_TLS_CLIENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: true,
},
KV {
key: MYSQL_QUEUE_DIR.to_owned(),
value: EVENT_DEFAULT_DIR.to_owned(),
hidden_if_empty: false,
},
KV {
key: MYSQL_QUEUE_LIMIT.to_owned(),
value: DEFAULT_LIMIT.to_string(),
hidden_if_empty: false,
},
KV {
key: MYSQL_MAX_OPEN_CONNECTIONS.to_owned(),
value: "2".to_owned(),
hidden_if_empty: false,
},
KV {
key: COMMENT_KEY.to_owned(),
value: "".to_owned(),
hidden_if_empty: false,
},
])
});
pub static DEFAULT_NOTIFY_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| mysql_kvs("rustfs_events"));
+67 -10
View File
@@ -246,16 +246,7 @@ impl Config {
}
let shard_size = shard_size as usize;
// Keep the historical two-data-shard object budget while preventing
// wider EC layouts from multiplying the maximum inline object size.
// Use div_ceil to match the shard_file_size calculation (which also uses
// div_ceil), avoiding a 1-byte rounding discrepancy that prevents inline
// for objects right at the threshold.
let inline_block = if self.initialized && self.inline_block_explicit {
self.inline_block
} else {
DEFAULT_INLINE_OBJECT_BUDGET.div_ceil(data_shards).min(DEFAULT_INLINE_BLOCK)
};
let inline_block = self.effective_inline_block(data_shards);
if versioned {
shard_size <= inline_block / 8
@@ -264,6 +255,27 @@ impl Config {
}
}
/// Returns the per-shard inline budget used by both write admission and
/// legacy read fallback.
///
/// The default budget is scaled by the number of data shards so a wider EC
/// layout does not silently increase the maximum inline object size. An
/// explicitly configured `inline_block` remains a fixed per-shard limit for
/// compatibility with deployments that opted into the historical policy.
pub(crate) fn effective_inline_block(&self, data_shards: usize) -> usize {
if data_shards == 0 {
return 0;
}
if self.initialized && self.inline_block_explicit {
self.inline_block
} else {
// Keep the historical two-data-shard object budget while preventing
// wider EC layouts from multiplying the maximum inline object size.
DEFAULT_INLINE_OBJECT_BUDGET.div_ceil(data_shards).min(DEFAULT_INLINE_BLOCK)
}
}
pub fn inline_block(&self) -> usize {
if !self.initialized {
DEFAULT_INLINE_BLOCK
@@ -602,6 +614,51 @@ mod tests {
}
}
#[test]
fn should_inline_keeps_ec8_and_ec12_object_boundaries_consistent() {
let config = Config::default();
let object_sizes = [128 * 1024, 256 * 1024, 512 * 1024, 1024 * 1024, 4 * 1024 * 1024];
for (data_shards, parity_shards) in [(8, 4), (12, 4)] {
let erasure = crate::erasure::coding::Erasure::new(data_shards, parity_shards, 1024 * 1024);
let mut previous = true;
for object_size in object_sizes {
let shard_size = erasure.shard_file_size(object_size);
let inline = config.should_inline(shard_size, data_shards, false);
// The effective policy is monotonic across object sizes. This
// table covers the boundaries that previously exposed the
// fixed-shard read-ahead mismatch, including the 1 MiB case.
assert!(!inline || previous, "inline decision must not re-enable at {object_size} bytes");
previous = inline;
}
assert!(
!config.should_inline(erasure.shard_file_size(1024 * 1024), data_shards, false),
"1 MiB must use the non-inline path for EC{data_shards}+{parity_shards}"
);
}
}
#[test]
fn effective_inline_block_scales_default_budget_and_preserves_explicit_limit() {
let config = Config::default();
assert_eq!(config.effective_inline_block(8), 32 * 1024);
assert_eq!(config.effective_inline_block(12), 21_846);
assert_eq!(config.effective_inline_block(0), 0);
let explicit = lookup_config_for_pools_with_env(
&KVS::new(),
&[12],
StorageClassEnvOverrides {
inline_block: Some("128KiB".to_string()),
..Default::default()
},
)
.expect("explicit inline block should resolve");
assert_eq!(explicit.effective_inline_block(12), 128 * 1024);
}
#[test]
fn explicit_inline_block_preserves_fixed_per_shard_rollback() {
let overrides = StorageClassEnvOverrides {
@@ -0,0 +1,414 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Shared default KVS tables for delivery targets that audit and notify declare identically.
//!
//! The audit and notify subsystems register one default KVS per delivery target. For amqp, nats,
//! pulsar, postgres and kafka both sides declare byte-identical tables; for redis and mysql they
//! differ only in a single default literal, which the caller passes in.
//!
//! Webhook and mqtt are deliberately absent: audit's webhook table carries extra batching/retry
//! keys and both tables disagree on key order and on several defaults (mqtt qos, keep-alive and
//! reconnect intervals), so they are real behavioral forks, not duplication.
//!
//! Key order is part of the contract: it drives the order admin config output lists the keys in,
//! so every constructor reproduces the existing order exactly.
use rustfs_config::server_config::{KV, KVS};
use rustfs_config::{
AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_SASL_ENABLE,
KAFKA_SASL_MECHANISM, KAFKA_SASL_PASSWORD, KAFKA_SASL_USERNAME, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY,
KAFKA_TLS_ENABLE, KAFKA_TOPIC, MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR,
MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS,
NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS, NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE,
NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT,
NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR,
POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY,
POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT,
PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL,
REDIS_CONNECTION_TIMEOUT, REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY,
REDIS_PASSWORD, REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS,
REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY,
REDIS_TLS_POLICY, REDIS_URL, REDIS_USERNAME,
};
/// Builds one default entry. `hidden_if_empty` marks values the admin API elides when unset.
fn kv(key: &str, value: impl Into<String>, hidden_if_empty: bool) -> KV {
KV {
key: key.to_owned(),
value: value.into(),
hidden_if_empty,
}
}
/// Default KVS for the amqp delivery target.
// Unused until the audit/notify tables are migrated onto these constructors.
#[allow(dead_code)]
pub fn amqp_kvs() -> KVS {
KVS(vec![
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
kv(AMQP_URL, "", false),
kv(AMQP_EXCHANGE, "", false),
kv(AMQP_ROUTING_KEY, "", false),
kv(AMQP_MANDATORY, EnableState::Off.to_string(), false),
kv(AMQP_PERSISTENT, EnableState::On.to_string(), false),
kv(AMQP_USERNAME, "", false),
kv(AMQP_PASSWORD, "", true),
kv(AMQP_TLS_CA, "", true),
kv(AMQP_TLS_CLIENT_CERT, "", true),
kv(AMQP_TLS_CLIENT_KEY, "", true),
kv(AMQP_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
kv(AMQP_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
kv(COMMENT_KEY, "", false),
])
}
/// Default KVS for the nats delivery target.
// Unused until the audit/notify tables are migrated onto these constructors.
#[allow(dead_code)]
pub fn nats_kvs() -> KVS {
KVS(vec![
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
kv(NATS_ADDRESS, "", false),
kv(NATS_SUBJECT, "", false),
kv(NATS_USERNAME, "", false),
kv(NATS_PASSWORD, "", true),
kv(NATS_TOKEN, "", true),
kv(NATS_CREDENTIALS_FILE, "", true),
kv(NATS_TLS_CA, "", true),
kv(NATS_TLS_CLIENT_CERT, "", true),
kv(NATS_TLS_CLIENT_KEY, "", true),
kv(NATS_TLS_REQUIRED, EnableState::Off.to_string(), false),
kv(NATS_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
kv(NATS_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
kv(NATS_JETSTREAM_ENABLE, EnableState::Off.to_string(), false),
kv(NATS_JETSTREAM_STREAM_NAME, "", false),
kv(
NATS_JETSTREAM_ACK_TIMEOUT_SECS,
NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(),
false,
),
kv(COMMENT_KEY, "", false),
])
}
/// Default KVS for the pulsar delivery target.
// Unused until the audit/notify tables are migrated onto these constructors.
#[allow(dead_code)]
pub fn pulsar_kvs() -> KVS {
KVS(vec![
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
kv(PULSAR_BROKER, "", false),
kv(PULSAR_TOPIC, "", false),
kv(PULSAR_AUTH_TOKEN, "", true),
kv(PULSAR_USERNAME, "", false),
kv(PULSAR_PASSWORD, "", true),
kv(PULSAR_TLS_CA, "", true),
kv(PULSAR_TLS_ALLOW_INSECURE, EnableState::Off.to_string(), false),
kv(PULSAR_TLS_HOSTNAME_VERIFICATION, EnableState::On.to_string(), false),
kv(PULSAR_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
kv(PULSAR_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
kv(COMMENT_KEY, "", false),
])
}
/// Default KVS for the postgres delivery target.
// Unused until the audit/notify tables are migrated onto these constructors.
#[allow(dead_code)]
pub fn postgres_kvs() -> KVS {
KVS(vec![
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
kv(POSTGRES_DSN_STRING, "", true),
kv(POSTGRES_TABLE, "", false),
kv(POSTGRES_FORMAT, "namespace", false),
kv(POSTGRES_TLS_REQUIRED, EnableState::Off.to_string(), false),
kv(POSTGRES_TLS_CA, "", true),
kv(POSTGRES_TLS_CLIENT_CERT, "", true),
kv(POSTGRES_TLS_CLIENT_KEY, "", true),
kv(POSTGRES_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
kv(POSTGRES_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
kv(COMMENT_KEY, "", false),
])
}
/// Default KVS for the kafka delivery target.
// Unused until the audit/notify tables are migrated onto these constructors.
#[allow(dead_code)]
pub fn kafka_kvs() -> KVS {
KVS(vec![
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
kv(KAFKA_BROKERS, "", false),
kv(KAFKA_TOPIC, "", false),
kv(KAFKA_ACKS, "1", false),
kv(KAFKA_TLS_ENABLE, EnableState::Off.to_string(), false),
kv(KAFKA_TLS_CA, "", true),
kv(KAFKA_TLS_CLIENT_CERT, "", true),
kv(KAFKA_TLS_CLIENT_KEY, "", true),
kv(KAFKA_SASL_ENABLE, EnableState::Off.to_string(), false),
kv(KAFKA_SASL_MECHANISM, "", false),
kv(KAFKA_SASL_USERNAME, "", false),
kv(KAFKA_SASL_PASSWORD, "", true),
kv(KAFKA_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
kv(KAFKA_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
kv(COMMENT_KEY, "", false),
])
}
/// Default KVS for the redis delivery target. `channel` is the subsystem's default pub/sub channel,
/// which is the only value audit and notify disagree on.
// Unused until the audit/notify tables are migrated onto these constructors.
#[allow(dead_code)]
pub fn redis_kvs(channel: &str) -> KVS {
KVS(vec![
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
kv(REDIS_URL, "", false),
kv(REDIS_CHANNEL, channel, false),
kv(REDIS_USERNAME, "", false),
kv(REDIS_PASSWORD, "", true),
kv(REDIS_KEEP_ALIVE_INTERVAL, "15", false),
kv(REDIS_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
kv(REDIS_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
kv(REDIS_MAX_RETRY_ATTEMPTS, "3", false),
kv(REDIS_RECONNECT_RETRY_ATTEMPTS, "", false),
kv(REDIS_MIN_RETRY_DELAY, "", false),
kv(REDIS_MAX_RETRY_DELAY, "", false),
kv(REDIS_CONNECTION_TIMEOUT, "", false),
kv(REDIS_RESPONSE_TIMEOUT, "", false),
kv(REDIS_PIPELINE_BUFFER_SIZE, "", false),
kv(REDIS_TLS_POLICY, "", true),
kv(REDIS_TLS_CA, "", true),
kv(REDIS_TLS_CLIENT_CERT, "", true),
kv(REDIS_TLS_CLIENT_KEY, "", true),
kv(REDIS_TLS_ALLOW_INSECURE, EnableState::Off.to_string(), false),
kv(COMMENT_KEY, "", false),
])
}
/// Default KVS for the mysql delivery target. `table` is the subsystem's default destination table,
/// which is the only value audit and notify disagree on.
// Unused until the audit/notify tables are migrated onto these constructors.
#[allow(dead_code)]
pub fn mysql_kvs(table: &str) -> KVS {
KVS(vec![
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
kv(MYSQL_DSN_STRING, "", true),
kv(MYSQL_TABLE, table, false),
kv(MYSQL_FORMAT, "access", false),
kv(MYSQL_TLS_CA, "", true),
kv(MYSQL_TLS_CLIENT_CERT, "", true),
kv(MYSQL_TLS_CLIENT_KEY, "", true),
kv(MYSQL_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
kv(MYSQL_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
kv(MYSQL_MAX_OPEN_CONNECTIONS, "2", false),
kv(COMMENT_KEY, "", false),
])
}
#[cfg(test)]
mod tests {
use super::*;
/// Expected values are spelled out as literals on purpose: they mirror the tables currently
/// declared in `audit.rs` and `notify.rs`, so a drift in key order or in any default breaks
/// the test instead of silently changing admin config output.
fn assert_table(actual: &KVS, expected: &[(&str, &str, bool)]) {
let actual: Vec<(&str, &str, bool)> = actual
.0
.iter()
.map(|kv| (kv.key.as_str(), kv.value.as_str(), kv.hidden_if_empty))
.collect();
assert_eq!(actual, expected);
}
const QUEUE_DIR: &str = "/opt/rustfs/events";
const QUEUE_LIMIT: &str = "100000";
#[test]
fn amqp_table_matches_audit_and_notify() {
assert_table(
&amqp_kvs(),
&[
("enable", "off", false),
("url", "", false),
("exchange", "", false),
("routing_key", "", false),
("mandatory", "off", false),
("persistent", "on", false),
("username", "", false),
("password", "", true),
("tls_ca", "", true),
("tls_client_cert", "", true),
("tls_client_key", "", true),
("queue_dir", QUEUE_DIR, false),
("queue_limit", QUEUE_LIMIT, false),
("comment", "", false),
],
);
}
#[test]
fn nats_table_matches_audit_and_notify() {
assert_table(
&nats_kvs(),
&[
("enable", "off", false),
("address", "", false),
("subject", "", false),
("username", "", false),
("password", "", true),
("token", "", true),
("credentials_file", "", true),
("tls_ca", "", true),
("tls_client_cert", "", true),
("tls_client_key", "", true),
("tls_required", "off", false),
("queue_dir", QUEUE_DIR, false),
("queue_limit", QUEUE_LIMIT, false),
("jetstream_enable", "off", false),
("jetstream_stream_name", "", false),
("jetstream_ack_timeout_secs", "30", false),
("comment", "", false),
],
);
}
#[test]
fn pulsar_table_matches_audit_and_notify() {
assert_table(
&pulsar_kvs(),
&[
("enable", "off", false),
("broker", "", false),
("topic", "", false),
("auth_token", "", true),
("username", "", false),
("password", "", true),
("tls_ca", "", true),
("tls_allow_insecure", "off", false),
("tls_hostname_verification", "on", false),
("queue_dir", QUEUE_DIR, false),
("queue_limit", QUEUE_LIMIT, false),
("comment", "", false),
],
);
}
#[test]
fn postgres_table_matches_audit_and_notify() {
assert_table(
&postgres_kvs(),
&[
("enable", "off", false),
("dsn_string", "", true),
("table", "", false),
("format", "namespace", false),
("tls_required", "off", false),
("tls_ca", "", true),
("tls_client_cert", "", true),
("tls_client_key", "", true),
("queue_dir", QUEUE_DIR, false),
("queue_limit", QUEUE_LIMIT, false),
("comment", "", false),
],
);
}
#[test]
fn kafka_table_matches_audit_and_notify() {
assert_table(
&kafka_kvs(),
&[
("enable", "off", false),
("brokers", "", false),
("topic", "", false),
("acks", "1", false),
("tls_enable", "off", false),
("tls_ca", "", true),
("tls_client_cert", "", true),
("tls_client_key", "", true),
("sasl_enable", "off", false),
("sasl_mechanism", "", false),
("sasl_username", "", false),
("sasl_password", "", true),
("queue_dir", QUEUE_DIR, false),
("queue_limit", QUEUE_LIMIT, false),
("comment", "", false),
],
);
}
fn expected_redis(channel: &str) -> Vec<(&str, &str, bool)> {
vec![
("enable", "off", false),
("url", "", false),
("channel", channel, false),
("username", "", false),
("password", "", true),
("keep_alive_interval", "15", false),
("queue_dir", QUEUE_DIR, false),
("queue_limit", QUEUE_LIMIT, false),
("max_retry_attempts", "3", false),
("reconnect_retry_attempts", "", false),
("min_retry_delay", "", false),
("max_retry_delay", "", false),
("connection_timeout", "", false),
("response_timeout", "", false),
("pipeline_buffer_size", "", false),
("tls_policy", "", true),
("tls_ca", "", true),
("tls_client_cert", "", true),
("tls_client_key", "", true),
("tls_allow_insecure", "off", false),
("comment", "", false),
]
}
#[test]
fn redis_table_matches_audit() {
assert_table(&redis_kvs("rustfs_audit_channel"), &expected_redis("rustfs_audit_channel"));
}
#[test]
fn redis_table_matches_notify() {
assert_table(&redis_kvs("rustfs_notify_channel"), &expected_redis("rustfs_notify_channel"));
}
fn expected_mysql(table: &str) -> Vec<(&str, &str, bool)> {
vec![
("enable", "off", false),
("dsn_string", "", true),
("table", table, false),
("format", "access", false),
("tls_ca", "", true),
("tls_client_cert", "", true),
("tls_client_key", "", true),
("queue_dir", QUEUE_DIR, false),
("queue_limit", QUEUE_LIMIT, false),
("max_open_connections", "2", false),
("comment", "", false),
]
}
#[test]
fn mysql_table_matches_audit() {
assert_table(&mysql_kvs("rustfs_audit_logs"), &expected_mysql("rustfs_audit_logs"));
}
#[test]
fn mysql_table_matches_notify() {
assert_table(&mysql_kvs("rustfs_events"), &expected_mysql("rustfs_events"));
}
}
+30 -2
View File
@@ -96,6 +96,8 @@ const LOG_SUBSYSTEM_POOLS: &str = "pools";
const EVENT_DECOMMISSION_STATE: &str = "decommission_state";
const EVENT_DECOMMISSION_BUCKET: &str = "decommission_bucket";
const EVENT_DECOMMISSION_ENTRY: &str = "decommission_entry";
const POOL_ACTIVATION_FLEET_PROOF_REQUIRED: &str = "pool activation requires a live fleet capability proof";
const POOL_ACTIVATION_FLEET_PROOF_EXPIRED: &str = "pool activation fleet capability proof expired before commit";
const DECOMMISSION_STAGE_MIGRATE_OBJECT: &str = "migrate_object";
const DECOMMISSION_STAGE_CLEANUP_PREFLIGHT: &str = "cleanup_preflight";
const DECOMMISSION_STAGE_SOURCE_CLEANUP: &str = "source_cleanup";
@@ -1832,6 +1834,13 @@ pub(crate) struct PoolRebalanceActivationFence {
}
impl PoolRebalanceActivationFence {
pub(crate) fn set_fleet_proof(
&mut self,
fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
) {
self.fleet_proof = fleet_proof;
}
pub(crate) fn ensure_held(&self) -> Result<()> {
#[cfg(test)]
let forced_lost = self.forced_lost.load(Ordering::Acquire);
@@ -1845,7 +1854,7 @@ impl PoolRebalanceActivationFence {
.as_ref()
.is_some_and(|proof| !crate::services::notification_sys::cross_pool_fence_fleet_proof_matches(proof))
{
return Err(Error::other("pool activation fleet capability proof expired before commit"));
return Err(Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED));
}
Ok(())
@@ -1901,7 +1910,17 @@ pub(crate) async fn acquire_pool_activation_fleet_proof(
}
crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof()
.map(Some)
.ok_or_else(|| Error::other("pool activation requires a live fleet capability proof"))
.ok_or_else(|| Error::other(POOL_ACTIVATION_FLEET_PROOF_REQUIRED))
}
pub(crate) fn is_pool_activation_fleet_proof_error(err: &Error) -> bool {
// Save-stage helpers add context by formatting the original error, so the
// marker may be nested in the display string. Restrict matching to the
// `Error::other` I/O shape used by this activation path.
matches!(err, Error::Io(io_error) if io_error.kind() == std::io::ErrorKind::Other && {
let message = io_error.to_string();
message.contains(POOL_ACTIVATION_FLEET_PROOF_REQUIRED) || message.contains(POOL_ACTIVATION_FLEET_PROOF_EXPIRED)
})
}
#[cfg(test)]
@@ -10828,6 +10847,15 @@ mod tests {
use crate::bucket::replication::{ReplicationState, ReplicationStatusType};
use serde::Serialize;
#[test]
fn pool_activation_fleet_proof_error_classifier_matches_only_retryable_proof_failures() {
assert!(is_pool_activation_fleet_proof_error(&Error::other(POOL_ACTIVATION_FLEET_PROOF_REQUIRED)));
assert!(is_pool_activation_fleet_proof_error(&Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED)));
let wrapped = format!("rebalance meta save failed during start_rebalance: {POOL_ACTIVATION_FLEET_PROOF_EXPIRED}");
assert!(is_pool_activation_fleet_proof_error(&Error::other(wrapped)));
assert!(!is_pool_activation_fleet_proof_error(&Error::ConfigNotFound));
}
#[tokio::test]
#[serial_test::serial]
async fn decommission_activation_fence_loss_after_durable_save_blocks_publication() {
@@ -15,7 +15,8 @@
use crate::error::{Error, Result};
use crate::runtime::sources::{self as runtime_sources, WorkloadSnapshotProviderRef};
use metrics::{counter, histogram};
use rustfs_concurrency::{AdmissionState, WorkloadAdmissionSnapshotProvider, WorkloadClass};
use rustfs_concurrency::WorkloadAdmissionSnapshotProvider;
use rustfs_concurrency::workload::ForegroundPressure;
use std::time::{Duration, Instant};
use tokio::time::sleep;
use tokio_util::sync::CancellationToken;
@@ -137,23 +138,6 @@ async fn wait_for_data_movement_admission_with_provider(
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
struct ForegroundPressure {
class: WorkloadClass,
usage_pct: usize,
threshold_pct: usize,
}
impl ForegroundPressure {
const fn reason(self) -> &'static str {
match self.class {
WorkloadClass::ForegroundRead => "foreground_read_pressure",
WorkloadClass::ForegroundWrite => "foreground_write_pressure",
_ => "foreground_pressure",
}
}
}
fn foreground_pressure(
config: &DataMovementBackpressureConfig,
provider: Option<&(dyn WorkloadAdmissionSnapshotProvider + Send + Sync)>,
@@ -163,39 +147,11 @@ fn foreground_pressure(
}
let snapshot = provider?.workload_admission_snapshot();
[
(WorkloadClass::ForegroundRead, config.foreground_read_high_percent),
(WorkloadClass::ForegroundWrite, config.foreground_write_high_percent),
]
.into_iter()
.filter_map(|(class, threshold_pct)| {
if threshold_pct == 0 {
return None;
}
let entry = snapshot.get(class)?;
let usage_pct = if matches!(entry.state, AdmissionState::Saturated) {
100
} else {
let limit = entry.limit?;
if limit == 0 {
return None;
}
entry
.active
.unwrap_or(0)
.saturating_mul(100)
.checked_div(limit)
.unwrap_or(100)
};
(usage_pct >= threshold_pct).then_some(ForegroundPressure {
class,
usage_pct,
threshold_pct,
})
})
.max_by_key(|pressure| pressure.usage_pct)
rustfs_concurrency::workload::foreground_pressure(
&snapshot,
config.foreground_read_high_percent,
config.foreground_write_high_percent,
)
}
fn record_delay_start(
@@ -276,7 +232,7 @@ fn record_delay_completion(
#[cfg(test)]
mod tests {
use super::*;
use rustfs_concurrency::{WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot};
use rustfs_concurrency::{AdmissionState, WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot, WorkloadClass};
use std::sync::Arc;
#[derive(Debug)]
+136
View File
@@ -1161,11 +1161,45 @@ fn select_admin_data_usage_snapshot(
authoritative.usage_snapshot_converged = Some(true);
}
match observed {
Some(observed)
if observed.usage_snapshot_partial
&& authoritative.is_complete_bucket_usage_snapshot()
&& observed_data_usage_is_newer(&observed, &authoritative) =>
{
(merge_partial_observation_for_admin(authoritative, observed), true)
}
Some(observed) if observed_data_usage_is_newer(&observed, &authoritative) => (observed, true),
_ => (authoritative, authoritative_format),
}
}
fn merge_partial_observation_for_admin(mut authoritative: DataUsageInfo, observed: DataUsageInfo) -> DataUsageInfo {
for (bucket, usage) in observed.buckets_usage {
authoritative.buckets_usage.insert(bucket, usage);
}
authoritative.last_update = observed.last_update;
authoritative.scanner_cycle = observed.scanner_cycle;
authoritative.scanner_epoch = observed.scanner_epoch;
authoritative.usage_snapshot_complete = false;
authoritative.usage_snapshot_partial = true;
authoritative.usage_snapshot_converged = Some(false);
authoritative.usage_snapshot_authoritative_baseline = observed.usage_snapshot_authoritative_baseline;
authoritative.usage_snapshot_set_states = observed.usage_snapshot_set_states;
authoritative.usage_snapshot_bootstrap_pending = false;
authoritative.buckets_count = authoritative.buckets_usage.len() as u64;
authoritative.bucket_sizes = authoritative
.buckets_usage
.iter()
.map(|(bucket, usage)| (bucket.clone(), usage.size))
.collect();
authoritative.replication_info.clear();
authoritative.tier_stats = None;
authoritative.unknown_tier_stats = None;
authoritative.calculate_totals();
authoritative
}
async fn load_admin_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
let (authoritative, source) = load_data_usage_snapshot(store.clone()).await?;
let observed = load_observed_data_usage_snapshot(store).await;
@@ -3217,6 +3251,108 @@ mod tests {
assert_eq!(selected.buckets_usage.get("bucket").map(|usage| usage.size), Some(100));
}
#[test]
fn partial_admin_observation_preserves_authoritative_cold_buckets() {
let baseline_time = SystemTime::UNIX_EPOCH + Duration::from_secs(10);
let mut authoritative = data_usage_info_for_test("cold", 152_318, 80 * 1024 * 1024 * 1024, baseline_time);
authoritative.scanner_epoch = Some(4);
authoritative.scanner_cycle = Some(10);
authoritative.buckets_usage.insert(
"hot".to_string(),
BucketUsageInfo {
objects_count: 3_000,
versions_count: 3_000,
size: 400 * 1024 * 1024,
..Default::default()
},
);
authoritative.buckets_count = 2;
authoritative.bucket_sizes = authoritative
.buckets_usage
.iter()
.map(|(bucket, usage)| (bucket.clone(), usage.size))
.collect();
authoritative.calculate_totals();
authoritative.replication_info.insert(
"stale-target".to_string(),
BucketTargetUsageInfo {
replicated_size: 400 * 1024 * 1024,
replicated_count: 3_000,
..Default::default()
},
);
authoritative.tier_stats = Some(rustfs_data_usage::AllTierStats {
tiers: HashMap::from([(
"WARM".to_string(),
rustfs_data_usage::TierStats {
total_size: 80 * 1024 * 1024 * 1024,
num_versions: 152_318,
num_objects: 152_318,
},
)]),
});
let mut observed = DataUsageInfo {
last_update: Some(baseline_time + Duration::from_secs(1)),
scanner_epoch: Some(4),
scanner_cycle: Some(11),
usage_snapshot_complete: false,
usage_snapshot_partial: true,
usage_snapshot_converged: Some(false),
usage_snapshot_authoritative_baseline: Some(authoritative.snapshot_identity()),
usage_snapshot_set_states: vec![rustfs_data_usage::DataUsageSnapshotSetState {
pool_index: 0,
set_index: 0,
scanner_cycle: Some(11),
scanner_epoch: Some(4),
scan_plan_digest: Some([1; 32]),
complete: true,
tombstone: false,
}],
..Default::default()
};
observed.buckets_usage.insert(
"hot".to_string(),
BucketUsageInfo {
objects_count: 34,
versions_count: 34,
size: 8 * 1024 * 1024,
..Default::default()
},
);
observed.buckets_count = 1;
observed.bucket_sizes.insert("hot".to_string(), 8 * 1024 * 1024);
observed.calculate_totals();
let (selected, current_format) = select_admin_data_usage_snapshot(authoritative, true, Some(observed));
assert!(current_format);
assert!(!selected.usage_snapshot_complete);
assert!(selected.usage_snapshot_partial);
assert!(selected.is_valid_partial_snapshot());
assert_eq!(selected.usage_snapshot_converged, Some(false));
assert_eq!(selected.buckets_count, 2);
assert_eq!(
selected
.buckets_usage
.get("cold")
.map(|usage| (usage.objects_count, usage.size)),
Some((152_318, 80 * 1024 * 1024 * 1024))
);
assert_eq!(
selected
.buckets_usage
.get("hot")
.map(|usage| (usage.objects_count, usage.size)),
Some((34, 8 * 1024 * 1024))
);
assert_eq!(selected.objects_total_count, 152_352);
assert_eq!(selected.objects_total_size, 80 * 1024 * 1024 * 1024 + 8 * 1024 * 1024);
assert!(selected.replication_info.is_empty());
assert!(selected.tier_stats.is_none());
assert!(selected.unknown_tier_stats.is_none());
}
#[tokio::test]
async fn authoritative_save_cleanup_removes_observed_snapshot_best_effort() {
let store = UsageCasStore::default();
+37
View File
@@ -22,6 +22,7 @@ pub type Error = DiskError;
pub type Result<T> = core::result::Result<T, Error>;
const METACACHE_OUTPUT_STREAM_CLOSED: &str = "metacache output stream closed";
pub(crate) const HEAL_DANGLING_DELETE_GRACE_MESSAGE: &str = "dangling object deletion deferred by heal grace window";
/// Marker carried by a shard-read `io::Error` when the underlying reader can
/// no longer be realigned after a fresh remote open failed. The marker is
@@ -33,6 +34,12 @@ pub(crate) struct TerminalReadError {
source: DiskError,
}
#[derive(Debug)]
struct DanglingDeleteGraceError {
retry_after_secs: i64,
grace_secs: i64,
}
// DiskError == StorageErr
#[derive(Debug, thiserror::Error)]
pub enum DiskError {
@@ -200,6 +207,18 @@ impl StdError for TerminalReadError {
}
}
impl std::fmt::Display for DanglingDeleteGraceError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(
f,
"{HEAL_DANGLING_DELETE_GRACE_MESSAGE}; retry_after_secs={}; grace_secs={}",
self.retry_after_secs, self.grace_secs
)
}
}
impl StdError for DanglingDeleteGraceError {}
fn classify_internode_missing_error(error: &InternodeHttpError) -> Option<DiskError> {
if error.is_remote_file_not_found() {
return Some(DiskError::FileNotFound);
@@ -253,6 +272,24 @@ impl DiskError {
DiskError::Io(std::io::Error::other(error))
}
pub(crate) fn dangling_delete_grace(retry_after_secs: i64, grace_secs: i64) -> Self {
DiskError::other(DanglingDeleteGraceError {
retry_after_secs,
grace_secs,
})
}
pub fn is_dangling_delete_grace(&self) -> bool {
matches!(self, DiskError::Io(io_error) if Self::io_error_is_dangling_delete_grace(io_error))
}
pub fn io_error_is_dangling_delete_grace(io_error: &io::Error) -> bool {
io_error
.get_ref()
.is_some_and(|source| source.downcast_ref::<DanglingDeleteGraceError>().is_some())
|| io_error.to_string().contains(HEAL_DANGLING_DELETE_GRACE_MESSAGE)
}
pub(crate) fn metacache_output_stream_closed() -> Self {
DiskError::Io(std::io::Error::new(std::io::ErrorKind::BrokenPipe, METACACHE_OUTPUT_STREAM_CLOSED))
}
+63 -3
View File
@@ -12,7 +12,6 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::config::storageclass::DEFAULT_INLINE_BLOCK;
use crate::crash_inject::{self, CrashPoint};
use crate::data_usage::local_snapshot::ensure_data_usage_layout;
use crate::diagnostics::get::{
@@ -10410,8 +10409,14 @@ impl DiskAPI for LocalDisk {
fi.data = None;
}
let inline = fi.transition_status.is_empty() && fi.data_dir.is_some() && fi.parts.len() == 1;
if inline && fi.shard_file_size(fi.parts[0].actual_size) < DEFAULT_INLINE_BLOCK as i64 {
// Keep this compatibility read-ahead decision on the same policy
// as PUT's inline admission. In particular, do not use the old
// fixed 128 KiB shard limit: a non-inline object in a wider EC
// layout can have a smaller shard and would otherwise be copied
// out of part.1 during every metadata read. Such objects remain
// fully readable through the normal EC reader below.
let storage_class_config = runtime_sources::storage_class_config_snapshot();
if should_read_legacy_inline_part(&fi, storage_class_config.as_ref()) {
let part_path = path_join_buf(&[
path,
fi.data_dir.map_or_else(|| "".to_string(), |dir| dir.to_string()).as_str(),
@@ -10913,6 +10918,21 @@ impl DiskAPI for LocalDisk {
}
}
/// Whether a legacy object without the inline marker should have its external
/// part materialized into `FileInfo.data` for compatibility with the old GET
/// fast path. The marker-bearing path is handled by `read_raw`/`get_file_info`;
/// this is only a conservative fallback for old metadata.
fn should_read_legacy_inline_part(fi: &FileInfo, storage_class_config: &crate::config::storageclass::Config) -> bool {
if !fi.transition_status.is_empty() || fi.data_dir.is_none() || fi.parts.len() != 1 || fi.inline_data() {
return false;
}
let part = &fi.parts[0];
let shard_size = fi.shard_file_size(part.actual_size);
let versioned = fi.versioned || fi.version_id.is_some_and(|version_id| !version_id.is_nil());
storage_class_config.should_inline(shard_size, fi.erasure.data_blocks, versioned)
}
impl LocalDisk {
pub(crate) async fn rename_data_borrowed(
&self,
@@ -11049,6 +11069,46 @@ mod test {
file_info
}
#[test]
fn legacy_inline_read_ahead_matches_writer_policy_for_ec_layouts() {
let config = crate::config::storageclass::Config::default();
let object_sizes = [128 * 1024_i64, 256 * 1024, 512 * 1024, 1024 * 1024, 4 * 1024 * 1024];
for (data_shards, parity_shards) in [(8, 4), (12, 4)] {
for object_size in object_sizes {
let mut fi = FileInfo::new("object", data_shards, parity_shards);
fi.data_dir = Some(Uuid::from_u128(1));
fi.parts = vec![ObjectPartInfo {
number: 1,
size: usize::try_from(object_size).expect("test object size should fit usize"),
actual_size: object_size,
..Default::default()
}];
let writer_decision = config.should_inline(fi.shard_file_size(object_size), data_shards, false);
assert_eq!(
should_read_legacy_inline_part(&fi, &config),
writer_decision,
"legacy read-ahead must match PUT for EC{data_shards}+{parity_shards}, size={object_size}"
);
}
}
let mut ec12 = FileInfo::new("object", 12, 4);
ec12.data_dir = Some(Uuid::from_u128(1));
ec12.parts = vec![ObjectPartInfo {
number: 1,
size: 1024 * 1024,
actual_size: 1024 * 1024,
..Default::default()
}];
assert!(
ec12.shard_file_size(1024 * 1024) < crate::config::storageclass::DEFAULT_INLINE_BLOCK as i64,
"the regression guard must exercise the old fixed 128 KiB read-ahead boundary"
);
assert!(!should_read_legacy_inline_part(&ec12, &config));
}
fn test_meta(fi: FileInfo) -> Vec<u8> {
let mut meta = FileMeta::default();
meta.add_version(fi).expect("test metadata should accept file info");
@@ -27,6 +27,8 @@ use std::io;
use std::io::ErrorKind;
use std::pin::Pin;
use std::sync::Mutex;
#[cfg(test)]
use std::sync::atomic::{AtomicU64, Ordering};
use std::task::{Context, Poll, ready};
use std::time::Instant;
use tokio::io::{AsyncRead, ReadBuf};
@@ -38,6 +40,14 @@ const DEFAULT_RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT: usize = 2;
const FILL_POLICY_SINGLE_INFLIGHT: &str = "single_inflight";
const FILL_POLICY_DUAL_INFLIGHT: &str = "dual_inflight";
#[cfg(test)]
static SINGLE_INFLIGHT_CONSTRUCTIONS: AtomicU64 = AtomicU64::new(0);
#[cfg(test)]
pub(crate) fn test_single_inflight_construction_count() -> u64 {
SINGLE_INFLIGHT_CONSTRUCTIONS.load(Ordering::Relaxed)
}
type FillTask = oneshot::Receiver<FillResult>;
struct FillWorker {
@@ -155,6 +165,23 @@ where
Self::new_with_fill_policy_inner(source, engine, total_length, metrics_path, FillPolicy::from_env())
}
/// Construct the bounded reader without lookahead.
///
/// Mid-size GETs are latency-sensitive and are already gated to a single
/// plain part. Keeping one stripe in flight avoids retaining a second
/// decoded output buffer while preserving the same source, reconstruction,
/// bitrot and cancellation semantics as the general streaming reader.
pub(crate) fn new_single_inflight_with_metrics_path(
source: S,
engine: E,
total_length: usize,
metrics_path: &'static str,
) -> io::Result<Self> {
#[cfg(test)]
SINGLE_INFLIGHT_CONSTRUCTIONS.fetch_add(1, Ordering::Relaxed);
Self::new_with_fill_policy_inner(source, engine, total_length, metrics_path, FillPolicy::SingleInFlight)
}
fn new_with_fill_policy_inner(
source: S,
engine: E,
@@ -602,7 +629,8 @@ where
loop {
if self.output_pos < self.output_buf.len() {
if self.prefetched_bufs.len() < self.fill_policy.max_inflight()
if self.fill_policy == FillPolicy::DualInFlight
&& self.prefetched_bufs.len() < self.fill_policy.max_inflight()
&& self.prefetch_error.is_none()
&& self.remaining > 0
&& let Poll::Ready(result) = self.poll_prefetch(cx)
@@ -1620,6 +1648,149 @@ mod tests {
assert_eq!(decoded, data);
}
#[tokio::test]
async fn single_inflight_reader_reads_full_body_without_lookahead() {
let erasure = Erasure::new(4, 2, 32);
let data = (0..96u8).collect::<Vec<_>>();
let read_count = Arc::new(AtomicUsize::new(0));
let mut source = source_from_data(&erasure, &data, &[]);
source.read_count = Some(Arc::clone(&read_count));
let engine = LegacyEcDecodeEngine::new(erasure);
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
source,
engine,
data.len(),
GET_OBJECT_PATH_CODEC_STREAMING,
)
.expect("single-inflight reader should be constructed");
let mut decoded = Vec::new();
reader
.read_to_end(&mut decoded)
.await
.expect("single-inflight reader should decode the complete body");
assert_eq!(decoded, data);
assert_eq!(
read_count.load(Ordering::SeqCst),
3,
"single-inflight must not read ahead after the final stripe"
);
}
#[tokio::test]
async fn single_inflight_reader_preserves_partial_reads() {
let erasure = Erasure::new(4, 2, 32);
let data = (0..83u8).collect::<Vec<_>>();
let engine = LegacyEcDecodeEngine::new(erasure.clone());
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
source_from_data(&erasure, &data, &[]),
engine,
data.len(),
GET_OBJECT_PATH_CODEC_STREAMING,
)
.expect("single-inflight reader should be constructed");
let mut decoded = Vec::with_capacity(data.len());
let mut chunk = [0u8; 3];
loop {
let read = reader.read(&mut chunk).await.expect("partial read should succeed");
if read == 0 {
break;
}
decoded.extend_from_slice(&chunk[..read]);
}
assert_eq!(decoded, data);
}
#[tokio::test]
async fn single_inflight_reader_does_not_prefetch_before_output_is_drained() {
let erasure = Erasure::new(4, 2, 32);
let data = (0..96u8).collect::<Vec<_>>();
let read_count = Arc::new(AtomicUsize::new(0));
let mut source = source_from_data(&erasure, &data, &[]);
source.read_count = Some(Arc::clone(&read_count));
let engine = LegacyEcDecodeEngine::new(erasure);
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
source,
engine,
data.len(),
GET_OBJECT_PATH_CODEC_STREAMING,
)
.expect("single-inflight reader should be constructed");
let mut first = [0u8; 3];
reader
.read_exact(&mut first)
.await
.expect("first partial read should succeed");
assert_eq!(
read_count.load(Ordering::SeqCst),
1,
"single-inflight must not prefetch while output remains"
);
assert_eq!(&first, &data[..3]);
}
#[tokio::test]
async fn single_inflight_reader_reconstructs_degraded_body() {
let erasure = Erasure::new(4, 2, 32);
let data = (0..97u16).map(|value| value as u8).collect::<Vec<_>>();
let engine = LegacyEcDecodeEngine::new(erasure.clone());
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
source_from_data(&erasure, &data, &[1]),
engine,
data.len(),
GET_OBJECT_PATH_CODEC_STREAMING,
)
.expect("single-inflight reader should be constructed");
let mut decoded = Vec::new();
reader
.read_to_end(&mut decoded)
.await
.expect("a readable degraded stripe should be reconstructed");
assert_eq!(decoded, data);
}
#[tokio::test]
async fn single_inflight_reader_surfaces_error_after_buffered_body() {
let erasure = Erasure::new(4, 2, 32);
let first_stripe = (0..32u8).collect::<Vec<_>>();
let first_state = source_from_data(&erasure, &first_stripe, &[])
.stripes
.pop_front()
.expect("first stripe should exist");
let source = VecStripeSource {
stripes: VecDeque::from([
first_state,
StripeReadState::from_parts(Vec::new(), Vec::new(), erasure.data_shards),
]),
read_quorum: erasure.data_shards,
read_count: None,
};
let engine = LegacyEcDecodeEngine::new(erasure);
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
source,
engine,
first_stripe.len() + 1,
GET_OBJECT_PATH_CODEC_STREAMING,
)
.expect("single-inflight reader should be constructed");
let mut decoded = Vec::new();
let error = reader
.read_to_end(&mut decoded)
.await
.expect_err("short source error should be returned after buffered bytes");
assert_eq!(error.kind(), ErrorKind::Other);
assert_eq!(decoded, first_stripe);
}
#[tokio::test]
async fn erasure_decode_reader_stops_at_eof_for_empty_object() {
let erasure = Erasure::new(4, 2, 32);
@@ -1882,14 +2053,9 @@ mod tests {
};
let engine = LegacyEcDecodeEngine::new(Erasure::new(1, 0, 32));
let task = tokio::spawn(async move {
let mut reader = ErasureDecodeReader::new_with_fill_policy(
source,
engine,
1,
GET_OBJECT_PATH_CODEC_STREAMING,
FillPolicy::SingleInFlight,
)
.expect("reader should be constructed");
let mut reader =
ErasureDecodeReader::new_single_inflight_with_metrics_path(source, engine, 1, GET_OBJECT_PATH_CODEC_STREAMING)
.expect("reader should be constructed");
let mut first_read = [0u8; 1];
let _ = reader.read(&mut first_read).await;
});
@@ -2226,7 +2392,7 @@ mod tests {
engine,
data.len(),
GET_OBJECT_PATH_CODEC_STREAMING,
FillPolicy::SingleInFlight,
FillPolicy::DualInFlight,
)
.expect("reader should be constructed");
let mut first_read = [0u8; 1];
@@ -2235,13 +2401,11 @@ mod tests {
assert_eq!(read, first_read.len());
assert_eq!(first_read[0], data[0]);
timeout(Duration::from_secs(1), async {
while read_count.load(Ordering::SeqCst) < 2 {
yield_now().await;
}
})
.await
.expect("reader should start reading the next stripe before the current output buffer is fully consumed");
assert_eq!(
read_count.load(Ordering::SeqCst),
2,
"dual-inflight reader should prefetch the next stripe before returning the first byte"
);
}
#[tokio::test]
+9
View File
@@ -157,6 +157,8 @@ pub enum StorageError {
InvalidPartNumber(usize),
#[error("Your proposed upload is smaller than the minimum allowed size. Part {0} size {1} is less than minimum {2}")]
EntityTooSmall(usize, i64, i64),
#[error("multipart upload size {0} exceeds the configured limit {1}")]
EntityTooLarge(u64, u64),
// ── Erasure / Quorum ─────────────────────────────────────────────
#[error("erasure read quorum")]
@@ -275,6 +277,10 @@ impl StorageError {
| StorageError::NamespaceLockQuorumUnavailable { .. }
)
}
pub fn is_dangling_delete_grace(&self) -> bool {
matches!(self, StorageError::Io(io_error) if DiskError::io_error_is_dangling_delete_grace(io_error))
}
}
impl From<HTTPRangeError> for StorageError {
@@ -554,6 +560,7 @@ impl Clone for StorageError {
StorageError::DecommissionNotStarted => StorageError::DecommissionNotStarted,
StorageError::InvalidPart(a, b, c) => StorageError::InvalidPart(*a, b.clone(), c.clone()),
StorageError::EntityTooSmall(a, b, c) => StorageError::EntityTooSmall(*a, *b, *c),
StorageError::EntityTooLarge(a, b) => StorageError::EntityTooLarge(*a, *b),
StorageError::DoneForNow => StorageError::DoneForNow,
StorageError::DecommissionAlreadyRunning => StorageError::DecommissionAlreadyRunning,
StorageError::RebalanceAlreadyRunning => StorageError::RebalanceAlreadyRunning,
@@ -673,6 +680,7 @@ impl StorageError {
StorageError::InsufficientWriteQuorum(_, _) => StorageErrorCode::InsufficientWriteQuorum,
StorageError::PreconditionFailed => StorageErrorCode::PreconditionFailed,
StorageError::EntityTooSmall(_, _, _) => StorageErrorCode::EntityTooSmall,
StorageError::EntityTooLarge(_, _) => StorageErrorCode::EntityTooLarge,
StorageError::InvalidRangeSpec(_) => StorageErrorCode::InvalidRangeSpec,
StorageError::NotModified => StorageErrorCode::NotModified,
StorageError::InvalidPartNumber(_) => StorageErrorCode::InvalidPartNumber,
@@ -795,6 +803,7 @@ impl StorageError {
StorageErrorCode::EntityTooSmall => {
Some(StorageError::EntityTooSmall(Default::default(), Default::default(), Default::default()))
}
StorageErrorCode::EntityTooLarge => Some(StorageError::EntityTooLarge(Default::default(), Default::default())),
StorageErrorCode::InvalidRangeSpec => Some(StorageError::InvalidRangeSpec(Default::default())),
StorageErrorCode::NotModified => Some(StorageError::NotModified),
StorageErrorCode::InvalidPartNumber => Some(StorageError::InvalidPartNumber(Default::default())),
+12 -1
View File
@@ -730,7 +730,18 @@ impl ReadPlan {
})
.await
.map_err(Error::other)?
.ok_or_else(|| Error::other("encrypted object metadata is incomplete"))?;
.ok_or_else(|| {
// The resolver saw no encryption it recognizes, yet the
// object's markers say it is encrypted. Keep failing closed,
// but as a typed, non-retryable error: the condition is a
// permanent property of the stored metadata, not a fault a
// retry can fix.
Error::other(EncryptionResolutionError::new(
EncryptionResolutionErrorKind::InvalidMetadata,
"object is marked encrypted, but no decryption material could be resolved from its metadata; \
the encryption metadata is incomplete or in a format this server cannot read",
))
})?;
let material = resolved;
#[cfg(feature = "rio-v2")]
let uses_legacy_encryption = matches!(material.mode, ReadEncryptionMode::Direct { .. });
+41 -19
View File
@@ -752,12 +752,18 @@ impl ObjectInfo {
.any(|key| rustfs_utils::http::is_object_encryption_marker(key))
}
/// Maximum inline size for non-versioned objects (128 KiB).
/// Matches `DEFAULT_INLINE_BLOCK` in `storageclass.rs`.
/// Historical non-versioned inline size reference.
///
/// Inline admission is layout-specific now; callers must not use this
/// constant to decide whether an object is eligible for the fast path.
#[deprecated(note = "inline eligibility is layout-specific; use persisted metadata and the read-path policy")]
pub const INLINE_MAX_SIZE: i64 = 128 * 1024;
/// Maximum inline size for versioned objects (16 KiB).
/// Matches `DEFAULT_INLINE_BLOCK / 8` in `storageclass.rs`.
/// Historical versioned inline size reference.
///
/// Inline admission is layout-specific now; callers must not use this
/// constant to decide whether an object is eligible for the fast path.
#[deprecated(note = "inline eligibility is layout-specific; use persisted metadata and the read-path policy")]
pub const INLINE_MAX_SIZE_VERSIONED: i64 = 16 * 1024;
/// Returns `true` when this object qualifies for the inline data fast path.
@@ -765,10 +771,12 @@ impl ObjectInfo {
/// The inline fast path decodes erasure-coded data entirely in memory,
/// bypassing disk I/O, duplex pipes, and the disk-read semaphore.
///
/// The `inlined` flag is the primary signal — PUT sets it through the
/// captured storage-class snapshot's `Config::should_inline`, which applies
/// the correct version-aware threshold (128 KiB non-versioned, 16 KiB versioned).
/// The size check below is a safety net using the same thresholds.
/// The persisted `inlined` flag is the canonical size-policy decision. PUT
/// sets it through the captured storage-class snapshot's effective policy,
/// which is layout- and version-aware. Reapplying a fixed object-size limit
/// here would disagree with that policy for wider EC layouts and explicit
/// inline configurations. The direct-memory reader retains its own bounded
/// 128 KiB allocation gate at the call site.
///
/// Additional conditions:
/// - Single part
@@ -779,14 +787,8 @@ impl ObjectInfo {
if !self.inlined {
return false;
}
// Apply the same version-aware threshold as PUT (storageclass.rs).
let max_size = if self.version_id.is_some() {
Self::INLINE_MAX_SIZE_VERSIONED
} else {
Self::INLINE_MAX_SIZE
};
self.parts.len() == 1
&& self.size <= max_size
&& self.size >= 0
&& !self.is_encrypted()
&& !self.is_compressed()
&& self.transitioned_object.tier.is_empty()
@@ -1382,14 +1384,14 @@ mod tests {
}
#[test]
fn inline_fast_path_eligibility_preserves_exact_versioned_boundaries() {
fn inline_fast_path_eligibility_follows_persisted_marker() {
for (case, size, versioned, expected) in [
("unversioned below", 128 * 1024 - 1, false, true),
("unversioned exact", 128 * 1024, false, true),
("unversioned above", 128 * 1024 + 1, false, false),
("unversioned above", 128 * 1024 + 1, false, true),
("versioned below", 16 * 1024 - 1, true, true),
("versioned exact", 16 * 1024, true, true),
("versioned above", 16 * 1024 + 1, true, false),
("versioned above", 16 * 1024 + 1, true, true),
] {
assert_eq!(
inline_fast_path_object(size, versioned).is_inline_fast_path_eligible(),
@@ -1399,9 +1401,29 @@ mod tests {
}
}
#[test]
fn inline_fast_path_marker_allows_ec8_and_ec12_layout_specific_256kib_objects() {
for data_blocks in [8, 12] {
let object = ObjectInfo {
size: 256 * 1024,
data_blocks,
parity_blocks: 4,
inlined: true,
version_id: Some(Uuid::from_u128(1)),
parts: Arc::new(vec![ObjectPartInfo::default()]),
..Default::default()
};
assert!(
object.is_inline_fast_path_eligible(),
"the persisted inline marker must be authoritative for EC{data_blocks}+4"
);
}
}
#[test]
fn inline_fast_path_eligibility_rejects_incompatible_object_shapes() {
let mut object = inline_fast_path_object(ObjectInfo::INLINE_MAX_SIZE, false);
let mut object = inline_fast_path_object(128 * 1024, false);
object.inlined = false;
assert!(!object.is_inline_fast_path_eligible(), "non-inline objects must fall back");
@@ -234,6 +234,39 @@ pub(crate) fn install_cross_pool_fence_fleet_proof_for_test() {
});
}
#[cfg(test)]
pub(crate) struct CrossPoolFenceFleetProofGuard {
previous_proof: Option<FleetCapabilityProof>,
previous_topology_conflict: bool,
}
#[cfg(test)]
impl Drop for CrossPoolFenceFleetProofGuard {
fn drop(&mut self) {
let mut state = cross_pool_fence_fleet_proof_slot()
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
state.proof = self.previous_proof.take();
state.topology_conflict = self.previous_topology_conflict;
}
}
/// Temporarily revoke the test proof so activation paths can exercise their
/// fail-closed behavior without changing the process-wide topology binding.
#[cfg(test)]
pub(crate) fn without_cross_pool_fence_fleet_proof_for_test() -> CrossPoolFenceFleetProofGuard {
let mut state = cross_pool_fence_fleet_proof_slot()
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let guard = CrossPoolFenceFleetProofGuard {
previous_proof: state.proof.clone(),
previous_topology_conflict: state.topology_conflict,
};
state.proof = None;
state.topology_conflict = true;
guard
}
#[cfg(any(test, feature = "test-util"))]
pub fn rotate_cross_pool_fence_fleet_proof_for_test() -> bool {
let mut state = cross_pool_fence_fleet_proof_slot()
@@ -570,10 +570,13 @@ impl ECStore {
where
S: EcstoreObjectIO + StorageNamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
{
let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
// Lock order: pool_meta_save_gate -> pool.bin -> rebalance.bin.
let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
pool_meta_guard.ensure_write_safe("rebalance worker activation")?;
let activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), fleet_proof).await?;
// Classify the durable rebalance record while holding both namespace
// fences. A terminal record is a no-op and must not depend on the
// notification subsystem having published a fleet proof yet.
let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?;
let pool_meta = self
.load_runtime_pool_meta_under_activation_fence(&mut pool_meta_guard, &activation_fence, "rebalance worker activation")
.await?;
@@ -597,10 +600,17 @@ impl ECStore {
}
activation_fence.ensure_held()?;
if !is_rebalance_conflicting_with_decommission(&persisted) {
if !crate::services::rebalance::rebalance_requires_worker_activation(&persisted) {
return Ok(RebalanceWorkerActivationFence::NotStartedTerminal);
}
// Active worker admission still requires the fail-closed fleet proof.
// Attach it immediately before the final fence validation so expiry or
// topology changes are checked again at every later commit boundary.
let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
activation_fence.set_fleet_proof(fleet_proof);
activation_fence.ensure_held()?;
Ok(RebalanceWorkerActivationFence::Ready(Box::new(activation_fence)))
}
@@ -1476,6 +1486,64 @@ mod tests {
assert_activation_locks_released(&store).await;
}
#[tokio::test]
#[serial_test::serial]
async fn rebalance_worker_skips_terminal_metadata_without_fleet_proof() {
let rebalance_id = "terminal-metadata-without-proof";
let completed = RebalanceMeta {
id: rebalance_id.to_string(),
pool_stats: vec![RebalanceStats {
participating: true,
info: RebalanceInfo {
status: RebalStatus::Completed,
..Default::default()
},
..Default::default()
}],
..Default::default()
};
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(Some(completed)).await;
let _proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
let activation = store
.fence_rebalance_worker_activation(store.pools[0].clone(), rebalance_id)
.await
.expect("terminal metadata should not require a fleet proof");
assert!(matches!(activation, RebalanceWorkerActivationFence::NotStartedTerminal));
}
#[tokio::test]
#[serial_test::serial]
async fn rebalance_worker_still_requires_fleet_proof_for_active_metadata() {
let rebalance_id = "active-metadata-without-proof";
let active = RebalanceMeta {
id: rebalance_id.to_string(),
pool_stats: vec![RebalanceStats {
participating: true,
info: RebalanceInfo {
status: RebalStatus::Started,
..Default::default()
},
..Default::default()
}],
..Default::default()
};
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(Some(active)).await;
let _proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
let err = match store
.fence_rebalance_worker_activation(store.pools[0].clone(), rebalance_id)
.await
{
Ok(_) => panic!("active metadata must not be admitted without a fleet proof"),
Err(err) => err,
};
assert!(
err.to_string()
.contains("pool activation requires a live fleet capability proof")
);
}
#[tokio::test]
#[serial_test::serial]
async fn rebalance_activation_adopts_commit_after_post_save_fence_loss() {
@@ -214,6 +214,14 @@ pub(super) fn is_rebalance_in_progress(meta: &RebalanceMeta) -> bool {
meta.pool_stats.iter().any(is_rebalance_pool_active)
}
/// Persisted rebalance metadata requires worker activation only while it has
/// not reached a durable terminal marker and at least one pool is still marked
/// active. Merely finding `rebalance.bin` is not evidence that admission is
/// required: terminal metadata is retained for status reporting.
pub(crate) fn rebalance_requires_worker_activation(meta: &RebalanceMeta) -> bool {
meta.stopped_at.is_none() && is_rebalance_in_progress(meta)
}
pub(crate) fn is_rebalance_conflicting_with_decommission(meta: &RebalanceMeta) -> bool {
is_rebalance_in_progress(meta)
}
+1 -1
View File
@@ -49,8 +49,8 @@ mod worker;
#[cfg(feature = "test-util")]
pub use entry::test_util::PausedRebalanceEntryTestFixture;
pub(crate) use meta::is_rebalance_conflicting_with_decommission;
pub use meta::{decode_rebalance_stop_propagation_record, encode_rebalance_stop_propagation_record};
pub(crate) use meta::{is_rebalance_conflicting_with_decommission, rebalance_requires_worker_activation};
pub use types::{
DiskStat, RebalSaveOpt, RebalStatus, RebalanceCleanupWarningEntry, RebalanceCleanupWarnings, RebalanceInfo, RebalanceMeta,
RebalanceStats, RebalanceStopPropagationRecord,
@@ -22,11 +22,11 @@ use super::meta::{
is_rebalance_in_progress, is_rebalance_meta_replaceable_for_new_id, is_rebalance_stopped_terminal_event,
mark_rebalance_bucket_done, merge_rebalance_bucket_lists, merge_rebalance_meta, next_rebal_bucket_from_stat,
percent_free_ratio, rebalance_goal_reached, rebalance_meta_load_no_data_error, rebalance_meta_load_unknown_format_error,
rebalance_meta_load_unknown_version_error, record_rebalance_cleanup_warning_in_meta, remove_rebalanced_buckets_from_queue,
resolve_next_rebalance_bucket, resolve_rebalance_participants, should_accept_rebalance_stats_update,
should_ignore_rebalance_data_usage_cache, should_pool_participate, should_preserve_rebalance_stopped_state,
should_skip_start_rebalance, stop_rebalance_meta_snapshot, stop_rebalance_state, take_bucket_from_rebalance_queue,
validate_init_rebalance_state, validate_start_rebalance_state,
rebalance_meta_load_unknown_version_error, rebalance_requires_worker_activation, record_rebalance_cleanup_warning_in_meta,
remove_rebalanced_buckets_from_queue, resolve_next_rebalance_bucket, resolve_rebalance_participants,
should_accept_rebalance_stats_update, should_ignore_rebalance_data_usage_cache, should_pool_participate,
should_preserve_rebalance_stopped_state, should_skip_start_rebalance, stop_rebalance_meta_snapshot, stop_rebalance_state,
take_bucket_from_rebalance_queue, validate_init_rebalance_state, validate_start_rebalance_state,
};
use super::migration::{
MigrationBackend, MigrationVersionResult, migrate_entry_version, migrate_entry_version_with_retry_wait,
@@ -3386,6 +3386,52 @@ fn test_is_rebalance_in_progress_only_started_participants() {
assert!(is_rebalance_in_progress(&meta));
}
#[test]
fn test_rebalance_requires_worker_activation_only_for_active_non_stopped_metadata() {
let now = OffsetDateTime::now_utc();
let active = RebalanceMeta {
pool_stats: vec![RebalanceStats {
participating: true,
info: RebalanceInfo {
status: RebalStatus::Started,
..Default::default()
},
..Default::default()
}],
..Default::default()
};
let stopped_active = RebalanceMeta {
stopped_at: Some(now),
pool_stats: active.pool_stats.clone(),
..Default::default()
};
assert!(rebalance_requires_worker_activation(&active));
for status in [
RebalStatus::Completed,
RebalStatus::Stopped,
RebalStatus::Failed,
RebalStatus::None,
] {
let terminal = RebalanceMeta {
pool_stats: vec![RebalanceStats {
participating: true,
info: RebalanceInfo {
status,
..Default::default()
},
..Default::default()
}],
..Default::default()
};
assert!(
!rebalance_requires_worker_activation(&terminal),
"terminal status {status:?} must not resume"
);
}
assert!(!rebalance_requires_worker_activation(&stopped_active));
}
#[test]
fn test_is_rebalance_conflicting_with_decommission_true_when_in_progress() {
let now = OffsetDateTime::now_utc();
+336 -24
View File
@@ -65,6 +65,7 @@ use crate::storage_api_contracts::{
};
use crate::{
bucket::lifecycle::{
get_lifecycle_config,
tier_delete_journal::{TIER_DELETE_JOURNAL_PREFIX, decode_tier_delete_journal_entry},
transition_transaction::{TRANSITION_TRANSACTION_RECORD_PREFIX, decode_transition_transaction_record},
},
@@ -81,7 +82,7 @@ use rustfs_filemeta::FileInfo;
use rustfs_rio::HashReader;
use rustfs_s3_client::{admin_handler_utils::AdminError, provider_versions::ProviderVersionCapabilities};
use rustfs_utils::path::{SLASH_SEPARATOR, path_join};
use s3s::S3ErrorCode;
use s3s::{S3ErrorCode, dto::BucketLifecycleConfiguration};
use super::{
tier_handlers::{ERR_TIER_BUCKET_NOT_FOUND, ERR_TIER_CONNECT_ERR, ERR_TIER_INVALID_CREDENTIALS, ERR_TIER_PERM_ERR},
@@ -535,6 +536,10 @@ enum TierCandidateMutation {
}
impl TierCandidateMutation {
fn checks_lifecycle_references(&self) -> bool {
!matches!(self, Self::Add(_, true) | Self::Remove(_, true) | Self::Clear(true))
}
fn intent_kind(&self) -> TierMutationIntentKind {
match self {
Self::Add(_, _) => TierMutationIntentKind::Add,
@@ -694,6 +699,7 @@ fn tier_backend_identity_admin_error(err: io::Error) -> AdminError {
admin_err
}
#[async_trait::async_trait]
trait TierReferenceProofStore:
EcstoreObjectIO
+ BucketOperations<Error = Error>
@@ -705,28 +711,27 @@ trait TierReferenceProofStore:
WalkOptions = TierReferenceProofWalkOptions,
WalkCancellation = tokio_util::sync::CancellationToken,
WalkResultSender = tokio::sync::mpsc::Sender<StorageObjectInfoOrErr<ObjectInfo, Error>>,
>
> + Send
+ Sync
{
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>>;
}
impl<T> TierReferenceProofStore for T where
T: EcstoreObjectIO
+ BucketOperations<Error = Error>
+ ListOperations<
Error = Error,
ListObjectsV2Info = StorageListObjectsV2Info<ObjectInfo>,
ListObjectVersionsInfo = StorageListObjectVersionsInfo<ObjectInfo>,
ObjectInfoOrErr = StorageObjectInfoOrErr<ObjectInfo, Error>,
WalkOptions = TierReferenceProofWalkOptions,
WalkCancellation = tokio_util::sync::CancellationToken,
WalkResultSender = tokio::sync::mpsc::Sender<StorageObjectInfoOrErr<ObjectInfo, Error>>,
>
{
#[async_trait::async_trait]
impl TierReferenceProofStore for ECStore {
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
match get_lifecycle_config(bucket).await {
Ok((config, _updated_at)) => Ok(Some(config)),
Err(Error::ConfigNotFound) => Ok(None),
Err(err) => Err(err),
}
}
}
async fn ensure_no_authoritative_tier_object_references<S>(
api: Arc<S>,
affected_targets: &[TierMutationIntentTarget],
check_lifecycle_references: bool,
) -> std::result::Result<(), AdminError>
where
S: TierReferenceProofStore,
@@ -739,12 +744,13 @@ where
if targets.is_empty() {
return Ok(());
}
ensure_no_authoritative_target_references(api, &targets).await
ensure_no_authoritative_target_references(api, &targets, check_lifecycle_references).await
}
async fn ensure_no_authoritative_target_references<S>(
api: Arc<S>,
targets: &[TierMutationIntentTarget],
check_lifecycle_references: bool,
) -> std::result::Result<(), AdminError>
where
S: TierReferenceProofStore,
@@ -754,6 +760,9 @@ where
.await
.map_err(tier_reference_proof_admin_error)?;
for bucket in buckets {
if check_lifecycle_references {
ensure_no_authoritative_lifecycle_references(api.as_ref(), &bucket.name, targets).await?;
}
let mut marker = None;
let mut version_marker = None;
loop {
@@ -789,6 +798,66 @@ where
ensure_no_authoritative_persisted_references(api, targets).await
}
async fn ensure_no_authoritative_lifecycle_references(
api: &impl TierReferenceProofStore,
bucket: &str,
targets: &[TierMutationIntentTarget],
) -> std::result::Result<(), AdminError> {
match api.lifecycle_config_for_reference_proof(bucket).await {
Ok(Some(config)) => {
if let Some(reference) = lifecycle_config_target_reference(&config, targets) {
return Err(tier_reference_proof_lifecycle_in_use_error(
reference.tier_name,
bucket,
reference.rule_id,
));
}
}
Ok(None) => {}
Err(err) => {
return Err(tier_reference_proof_admin_error(format!("bucket {bucket} lifecycle config: {err}")));
}
}
Ok(())
}
struct TierLifecycleReference<'a> {
tier_name: &'a str,
rule_id: Option<&'a str>,
}
fn lifecycle_config_target_reference<'a>(
config: &'a BucketLifecycleConfiguration,
targets: &[TierMutationIntentTarget],
) -> Option<TierLifecycleReference<'a>> {
for rule in &config.rules {
let rule_id = rule.id.as_deref();
if let Some(transitions) = &rule.transitions {
for transition in transitions {
let Some(storage_class) = &transition.storage_class else {
continue;
};
let tier_name = storage_class.as_str();
if !tier_name.is_empty() && targets.iter().any(|target| target.tier_name == tier_name) {
return Some(TierLifecycleReference { tier_name, rule_id });
}
}
}
if let Some(noncurrent_version_transitions) = &rule.noncurrent_version_transitions {
for transition in noncurrent_version_transitions {
let Some(storage_class) = &transition.storage_class else {
continue;
};
let tier_name = storage_class.as_str();
if !tier_name.is_empty() && targets.iter().any(|target| target.tier_name == tier_name) {
return Some(TierLifecycleReference { tier_name, rule_id });
}
}
}
}
None
}
async fn ensure_no_authoritative_persisted_references<S>(
api: Arc<S>,
targets: &[TierMutationIntentTarget],
@@ -920,6 +989,15 @@ fn tier_reference_proof_persisted_in_use_error(tier_name: &str, object: &str) ->
err
}
fn tier_reference_proof_lifecycle_in_use_error(tier_name: &str, bucket: &str, rule_id: Option<&str>) -> AdminError {
let mut err = ERR_TIER_BACKEND_IN_USE.clone();
err.message = match rule_id {
Some(rule_id) => format!("Remote tier {tier_name} is still referenced by lifecycle rule {rule_id} in bucket {bucket}"),
None => format!("Remote tier {tier_name} is still referenced by a lifecycle rule in bucket {bucket}"),
};
err
}
fn tier_reference_proof_admin_error(err: impl std::fmt::Display) -> AdminError {
let mut admin_err = ERR_TIER_INVALID_CONFIG.clone();
admin_err.message = format!("Remote tier reference proof failed: {err}");
@@ -2475,6 +2553,24 @@ impl TierConfigMgr {
})?;
}
// The Azure warm backend goes through the same S3-compatible TransitionClient as every
// other provider (backlog#2055): it has no Azure Blob-native client and no Azure AD
// dependency, so `storage_class` and `sp_auth` cannot be honored today even though the
// config type carries them. Reject them explicitly here instead of silently accepting
// and then dropping them at the WarmBackendAzure construction boundary.
if matches!(&tier_config.tier_type, TierType::Azure)
&& let Some(azure) = tier_config.azure.as_ref()
{
let sp_auth_set = !azure.sp_auth.tenant_id.is_empty()
|| !azure.sp_auth.client_id.is_empty()
|| !azure.sp_auth.client_secret.is_empty();
if !azure.storage_class.is_empty() || sp_auth_set {
let mut err = ERR_TIER_INVALID_CONFIG.clone();
err.message = "Azure remote tiers do not support storageClass or spAuth yet; leave both unset".to_string();
return Err(err);
}
}
let d = new_warm_backend(&tier_config, true).await?;
if !force {
@@ -3231,6 +3327,7 @@ impl TierConfigMgr {
let _config_lock = config_lock;
let _update = update;
let mutation_kind = mutation.intent_kind();
let check_lifecycle_references = mutation.checks_lifecycle_references();
if version.is_none() && !candidate.tiers.is_empty() && mutation_kind != TierMutationIntentKind::Add {
return Err(TierConfigUpdateError::Load(io::Error::other(
"tier configuration mutation requires an existing config ETag",
@@ -3269,7 +3366,7 @@ impl TierConfigMgr {
let affected_targets =
build_tier_mutation_affected_targets(mutation_kind, proof_targets, &current_for_targets, &candidate)
.map_err(TierConfigUpdateError::Publish)?;
ensure_no_authoritative_tier_object_references(api.clone(), &affected_targets)
ensure_no_authoritative_tier_object_references(api.clone(), &affected_targets, check_lifecycle_references)
.await
.map_err(TierConfigUpdateError::Publish)?;
let coordinator_intent =
@@ -5308,6 +5405,10 @@ mod tests {
endpoints::{Endpoints, PoolEndpoints, SetupType},
};
use crate::services::tier::tier_mutation_intent::TIER_MUTATION_INTENT_RECORD_PREFIX;
use s3s::dto::{
BucketLifecycleConfiguration, ExpirationStatus, LifecycleRule, NoncurrentVersionTransition, Transition,
TransitionStorageClass,
};
struct SetupTypeGuard {
previous: SetupType,
@@ -6093,6 +6194,13 @@ mod tests {
}
}
#[async_trait::async_trait]
impl TierReferenceProofStore for LockingTierConfigStore {
async fn lifecycle_config_for_reference_proof(&self, _bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
Ok(None)
}
}
#[tokio::test]
async fn tier_config_update_path_acquires_meta_namespace_sidecar_lock_before_save() {
let manager = TierConfigMgr::new();
@@ -6143,6 +6251,17 @@ mod tests {
assert!(noop_targets.is_empty());
}
#[test]
fn forced_tier_mutations_skip_only_lifecycle_reference_checks() {
assert!(!TierCandidateMutation::Add(build_rustfs_tier("COLD-A"), true).checks_lifecycle_references());
assert!(TierCandidateMutation::Add(build_rustfs_tier("COLD-A"), false).checks_lifecycle_references());
assert!(!TierCandidateMutation::Remove("COLD-A".to_string(), true).checks_lifecycle_references());
assert!(TierCandidateMutation::Remove("COLD-A".to_string(), false).checks_lifecycle_references());
assert!(!TierCandidateMutation::Clear(true).checks_lifecycle_references());
assert!(TierCandidateMutation::Clear(false).checks_lifecycle_references());
assert!(TierCandidateMutation::Edit("COLD-A".to_string(), TierCreds::default()).checks_lifecycle_references());
}
/// A fully offline `WarmBackend` used to exercise the driver-facing
/// branches of `remove`/`verify` without touching a remote tier.
struct MockWarmBackend {
@@ -6303,6 +6422,57 @@ mod tests {
assert!(mgr.tiers.is_empty());
}
#[tokio::test]
async fn test_add_rejects_azure_storage_class_before_backend_setup() {
let mut mgr = empty_mgr();
let mut tier = build_azure_tier("account-a");
tier.azure.as_mut().expect("Azure payload should exist").storage_class = "HOT".to_string();
let err = mgr
.add(tier, true)
.await
.expect_err("a non-empty Azure storageClass must be rejected before backend setup");
assert_eq!(err.code, ERR_TIER_INVALID_CONFIG.code);
assert!(err.message.contains("storageClass"), "{}", err.message);
assert!(mgr.tiers.is_empty());
}
#[tokio::test]
async fn test_add_rejects_azure_partial_sp_auth_before_backend_setup() {
// Only `tenant_id` is set: `TierAzure::is_sp_enabled()`-style "all three fields"
// logic would miss this, so the check must reject on *any* sp_auth sub-field
// being non-empty rather than requiring all three.
let mut mgr = empty_mgr();
let mut tier = build_azure_tier("account-a");
tier.azure.as_mut().expect("Azure payload should exist").sp_auth.tenant_id = "tenant".to_string();
let err = mgr
.add(tier, true)
.await
.expect_err("a partially-filled Azure spAuth must be rejected before backend setup");
assert_eq!(err.code, ERR_TIER_INVALID_CONFIG.code);
assert!(err.message.contains("spAuth"), "{}", err.message);
assert!(mgr.tiers.is_empty());
}
#[tokio::test]
async fn test_add_does_not_reject_azure_config_without_storage_class_or_sp_auth() {
// A plain Azure config (the common case: static access/secret key, no
// storageClass, no spAuth) must sail past the new gate. `new_warm_backend`
// builds the S3-compatible client lazily (no eager DNS/connect), so with
// `force: true` (which also skips the `in_use` probe) this succeeds even
// against a fake endpoint — the point here is only that the gate itself
// does not fire.
let mut mgr = empty_mgr();
let tier = build_azure_tier("account-a");
let tier_name = tier.name.clone();
mgr.add(tier, true)
.await
.expect("a config with no storageClass/spAuth must not trip the new gate");
assert!(mgr.tiers.contains_key(&tier_name));
}
#[tokio::test]
async fn test_add_rejects_reserved_names() {
// Supersedes the former `test_add_does_not_reserve_standard_name_regression_anchor`
@@ -11296,6 +11466,7 @@ mod tests {
lock_manager: Arc<rustfs_lock::GlobalLockManager>,
lock_requests: Mutex<Vec<(String, String)>>,
listed_versions: Mutex<Vec<ObjectInfo>>,
lifecycle_configs: Mutex<HashMap<String, BucketLifecycleConfiguration>>,
}
impl Default for CasConfigStore {
@@ -11316,6 +11487,7 @@ mod tests {
lock_manager: Arc::new(rustfs_lock::GlobalLockManager::new()),
lock_requests: Mutex::new(Vec::new()),
listed_versions: Mutex::new(Vec::new()),
lifecycle_configs: Mutex::new(HashMap::new()),
}
}
}
@@ -11342,6 +11514,13 @@ mod tests {
.push(object);
}
fn add_lifecycle_config(&self, bucket: &str, config: BucketLifecycleConfiguration) {
self.lifecycle_configs
.lock()
.expect("tier reference fixture should not poison")
.insert(bucket.to_string(), config);
}
async fn insert_config_object(&self, object: String, data: Vec<u8>) {
self.objects
.lock()
@@ -11917,9 +12096,46 @@ mod tests {
}
}
#[async_trait::async_trait]
impl TierReferenceProofStore for CasConfigStore {
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
Ok(self
.lifecycle_configs
.lock()
.expect("tier reference fixture should not poison")
.get(bucket)
.cloned())
}
}
#[tokio::test]
async fn remove_and_clear_full_update_paths_preserve_force() {
let lifecycle_config = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("force-remove".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: Some(vec![Transition {
days: Some(1),
date: None,
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
}]),
}],
};
let remove_store = Arc::new(CasConfigStore::default());
remove_store.add_listed_version(ObjectInfo {
bucket: "photos".to_string(),
name: "safe.txt".to_string(),
..Default::default()
});
remove_store.add_lifecycle_config("photos", lifecycle_config.clone());
let mut persisted = empty_mgr();
persisted.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
persisted
@@ -11946,6 +12162,12 @@ mod tests {
);
let clear_store = Arc::new(CasConfigStore::default());
clear_store.add_listed_version(ObjectInfo {
bucket: "photos".to_string(),
name: "safe.txt".to_string(),
..Default::default()
});
clear_store.add_lifecycle_config("photos", lifecycle_config);
persisted
.save_tiering_config_if_current(clear_store.clone(), None)
.await
@@ -12073,7 +12295,7 @@ mod tests {
"COLD-A",
Some(replacement_identity),
));
ensure_no_authoritative_tier_object_references(store.clone(), std::slice::from_ref(&target))
ensure_no_authoritative_tier_object_references(store.clone(), std::slice::from_ref(&target), true)
.await
.expect("references already written with the new destination identity should not block rebind");
@@ -12083,13 +12305,103 @@ mod tests {
"COLD-A",
Some(current_identity),
));
let err = ensure_no_authoritative_tier_object_references(store, &[target])
let err = ensure_no_authoritative_tier_object_references(store, &[target], true)
.await
.expect_err("references to the old destination identity must block rebind");
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
assert!(err.message.contains("photos/2026/old-destination.jpg"), "{}", err.message);
}
#[tokio::test]
async fn zero_reference_proof_blocks_lifecycle_transition_references() {
let current = build_rustfs_tier("COLD-A");
let current_identity = tier_backend_identity(&current).expect("current identity should encode");
let target = TierMutationIntentTarget {
tier_name: "COLD-A".to_string(),
old_backend_identity: Some(current_identity),
new_backend_identity: None,
};
let config = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("move-current".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: Some(vec![Transition {
days: Some(1),
date: None,
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
}]),
}],
};
let store = Arc::new(CasConfigStore::default());
store.add_listed_version(ObjectInfo {
bucket: "photos".to_string(),
name: "safe.txt".to_string(),
..Default::default()
});
store.add_lifecycle_config("photos", config);
let err = ensure_no_authoritative_tier_object_references(store, &[target], true)
.await
.expect_err("lifecycle rule should block deletion");
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
assert!(err.message.contains("move-current"), "{}", err.message);
assert!(err.message.contains("photos"), "{}", err.message);
}
#[tokio::test]
async fn zero_reference_proof_blocks_lifecycle_noncurrent_transition_references() {
let current = build_rustfs_tier("COLD-A");
let current_identity = tier_backend_identity(&current).expect("current identity should encode");
let target = TierMutationIntentTarget {
tier_name: "COLD-A".to_string(),
old_backend_identity: Some(current_identity),
new_backend_identity: None,
};
let config = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("move-noncurrent".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: Some(vec![NoncurrentVersionTransition {
noncurrent_days: Some(1),
newer_noncurrent_versions: None,
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
}]),
prefix: None,
transitions: None,
}],
};
let store = Arc::new(CasConfigStore::default());
store.add_listed_version(ObjectInfo {
bucket: "photos".to_string(),
name: "safe.txt".to_string(),
..Default::default()
});
store.add_lifecycle_config("photos", config);
let err = ensure_no_authoritative_tier_object_references(store, &[target], true)
.await
.expect_err("lifecycle rule should block deletion");
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
assert!(err.message.contains("move-noncurrent"), "{}", err.message);
assert!(err.message.contains("photos"), "{}", err.message);
}
#[tokio::test]
async fn zero_reference_proof_blocks_persisted_journal_transaction_and_free_version_references() {
let current = build_rustfs_tier("COLD-A");
@@ -12120,7 +12432,7 @@ mod tests {
.expect("journal record should encode"),
)
.await;
let err = ensure_no_authoritative_tier_object_references(journal_store, std::slice::from_ref(&target))
let err = ensure_no_authoritative_tier_object_references(journal_store, std::slice::from_ref(&target), true)
.await
.expect_err("unfinished delete journal for the old backend must block rebind");
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
@@ -12158,7 +12470,7 @@ mod tests {
transaction.encode().expect("transaction record should encode"),
)
.await;
let err = ensure_no_authoritative_tier_object_references(transaction_store, std::slice::from_ref(&target))
let err = ensure_no_authoritative_tier_object_references(transaction_store, std::slice::from_ref(&target), true)
.await
.expect_err("unfinished transition transaction for the old backend must block rebind");
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
@@ -12168,11 +12480,11 @@ mod tests {
let mut free_version = transitioned_tier_object("photos", "2026/free-version.jpg", "COLD-A", Some(current_identity));
free_version.transitioned_object.status = "pending".to_string();
free_version.transitioned_object.free_version = true;
ensure_no_authoritative_tier_object_references(free_version_store.clone(), std::slice::from_ref(&target))
ensure_no_authoritative_tier_object_references(free_version_store.clone(), std::slice::from_ref(&target), true)
.await
.expect("empty free-version fixture should permit rebind");
free_version_store.add_listed_version(free_version);
let err = ensure_no_authoritative_tier_object_references(free_version_store, &[target])
let err = ensure_no_authoritative_tier_object_references(free_version_store, &[target], true)
.await
.expect_err("recoverable free version for the old backend must block rebind");
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
@@ -12198,7 +12510,7 @@ mod tests {
}
store.omit_truncated_reference_marker();
let err = ensure_no_authoritative_tier_object_references(store, &[target])
let err = ensure_no_authoritative_tier_object_references(store, &[target], true)
.await
.expect_err("truncated authoritative reference scan without a marker must fail closed");
assert_eq!(err.code, ERR_TIER_INVALID_CONFIG.code);
@@ -646,12 +646,6 @@ pub struct TierAzure {
pub sp_auth: ServicePrincipalAuth,
}
impl TierAzure {
pub fn is_sp_enabled(&self) -> bool {
!self.sp_auth.tenant_id.is_empty() && !self.sp_auth.client_id.is_empty() && !self.sp_auth.client_secret.is_empty()
}
}
/*
fn AzureServicePrincipal(tenantID, clientID, clientSecret string) func(az *TierAzure) error {
return func(az *TierAzure) error {
@@ -36,11 +36,14 @@ use crate::services::tier::{
};
use bytes::Bytes;
use http::StatusCode;
use rustfs_s3_client::credentials::{Credentials, SignatureType, Static, Value};
use rustfs_s3_client::transition_api::{BucketLookupType, Options, TransitionClient, TransitionCore};
use rustfs_s3_client::{
admin_handler_utils::AdminError,
api_put_object::{AdvancedPutOptions, PutObjectOptions},
transition_api::{ReadCloser, ReaderImpl},
};
use rustfs_utils::egress::validate_outbound_url;
use rustfs_utils::http::headers::{
CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE, EXPIRES, HeaderExt as _,
};
@@ -50,6 +53,7 @@ use s3s::header::{
X_AMZ_STORAGE_CLASS,
};
use std::collections::HashMap;
use std::sync::Arc;
use time::OffsetDateTime;
use time::format_description::well_known::{Rfc2822, Rfc3339};
use tracing::{info, warn};
@@ -58,6 +62,11 @@ pub type WarmBackendImpl = Box<dyn WarmBackend + Send + Sync + 'static>;
const PROBE_OBJECT: &str = "probeobject";
/// Largest object the S3-compatible warm backends accept for a multipart put.
pub(crate) const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
/// Part-count ceiling S3-compatible services impose on a multipart upload.
pub(crate) const MAX_PARTS_COUNT: i64 = 10000;
#[derive(Default)]
pub struct WarmBackendGetOpts {
pub start_offset: i64,
@@ -222,6 +231,121 @@ pub fn build_transition_put_options(storage_class: String, mut metadata: HashMap
opts
}
/// Connection parameters every S3-compatible warm backend provider supplies.
///
/// The Aliyun, Azure, Huaweicloud, Tencent, MinIO, R2, and RustFS backends all
/// wrap [`WarmBackendS3`] around a statically-credentialed [`TransitionClient`]
/// built from exactly these values. `bucket_lookup` is a parameter rather than a
/// constant because the providers split into two families: Aliyun, Azure,
/// Huaweicloud, and Tencent pin [`BucketLookupType::BucketLookupDNS`], while
/// MinIO, R2, and RustFS leave it at [`BucketLookupType::BucketLookupAuto`].
pub(crate) struct S3CompatibleWarmBackendParams<'a> {
pub endpoint: &'a str,
pub access_key: &'a str,
pub secret_key: &'a str,
pub bucket: &'a str,
pub prefix: &'a str,
pub region: &'a str,
pub bucket_lookup: BucketLookupType,
/// Tag handed to [`TransitionClient::new`] so per-provider client behavior
/// and metrics stay attributable.
pub provider_tag: &'a str,
/// SSRF guard run against the parsed endpoint once it's known to have a
/// host. Almost every provider passes [`rustfs_utils::egress::validate_outbound_url`]
/// unchanged; RustFS passes its own wrapper that adds a debug-only,
/// env-gated loopback exception for its e2e tier tests (see
/// rustfs/rustfs#6773) — the shared constructor stays the single call
/// site either way, so no provider can silently end up unvalidated.
pub validate_endpoint: fn(&url::Url) -> Result<(), rustfs_utils::egress::OutboundUrlError>,
}
/// Build the [`WarmBackendS3`] shared by the S3-compatible warm backend providers.
///
/// Credential, bucket, and endpoint validation run in this order because the
/// existing provider constructors report the first failure they hit, and their
/// error texts are user-visible through the tier admin API.
pub(crate) async fn new_s3_compatible_warm_backend(
params: S3CompatibleWarmBackendParams<'_>,
) -> Result<WarmBackendS3, std::io::Error> {
if params.access_key.is_empty() || params.secret_key.is_empty() {
return Err(std::io::Error::other("both access and secret keys are required"));
}
if params.bucket.is_empty() {
return Err(std::io::Error::other("no bucket name was provided"));
}
let u = match url::Url::parse(params.endpoint) {
Ok(u) => u,
Err(e) => {
return Err(std::io::Error::other(e.to_string()));
}
};
let creds = Credentials::new(Static(Value {
access_key_id: params.access_key.to_string(),
secret_access_key: params.secret_key.to_string(),
session_token: "".to_string(),
signer_type: SignatureType::SignatureV4,
..Default::default()
}));
let opts = Options {
creds,
secure: u.scheme() == "https",
trailing_headers: true,
region: params.region.to_string(),
bucket_lookup: params.bucket_lookup,
..Default::default()
};
let scheme = u.scheme();
let default_port = if scheme == "https" { 443 } else { 80 };
let host = u
.host_str()
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
// Runs after the host-presence check above (not immediately after Url::parse) so a
// host-less endpoint still reports this constructor's own "missing host" text instead of
// validate_endpoint's differently-worded rejection for the same input.
(params.validate_endpoint)(&u).map_err(|err| std::io::Error::other(format!("tier endpoint is not allowed: {err}")))?;
let client =
TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, params.provider_tag).await?;
let client = Arc::new(client);
let core = TransitionCore(Arc::clone(&client));
Ok(WarmBackendS3 {
client,
core,
bucket: params.bucket.to_string(),
prefix: params.prefix.strip_suffix("/").unwrap_or(params.prefix).to_owned(),
storage_class: "".to_string(),
})
}
/// Round the multipart part size up to a whole multiple of `min_part_size` that
/// keeps the upload within [`MAX_PARTS_COUNT`] parts.
///
/// `object_size == -1` means "length unknown", so the caller is charged the
/// worst case of a full [`MAX_MULTIPART_PUT_OBJECT_SIZE`] object.
pub(crate) fn optimal_part_size(object_size: i64, min_part_size: i64) -> Result<i64, std::io::Error> {
let mut object_size = object_size;
if object_size == -1 {
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
}
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
return Err(std::io::Error::other("entity too large"));
}
let configured_part_size = min_part_size;
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
part_size_flt = (part_size_flt / configured_part_size as f64).ceil() * configured_part_size as f64;
let part_size = part_size_flt as i64;
if part_size == 0 {
return Ok(min_part_size);
}
Ok(part_size)
}
pub async fn check_warm_backend(w: Option<&WarmBackendImpl>) -> Result<(), AdminError> {
let w = w.ok_or_else(|| ERR_TIER_NOT_FOUND.clone())?;
w.validate().await.map_err(|_| ERR_TIER_INVALID_CONFIG.clone())?;
@@ -803,6 +927,194 @@ mod tests {
assert_eq!(err.code, ERR_TIER_INVALID_CONFIG.code);
}
/// Every S3-compatible provider file pins this same floor today.
const PROVIDER_MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
fn s3_compatible_params(endpoint: &str) -> S3CompatibleWarmBackendParams<'_> {
S3CompatibleWarmBackendParams {
endpoint,
access_key: "access",
secret_key: "secret",
bucket: "tier-bucket",
prefix: "archive",
region: "us-east-1",
bucket_lookup: BucketLookupType::BucketLookupDNS,
provider_tag: "aliyun",
validate_endpoint: validate_outbound_url,
}
}
/// `WarmBackendS3` has no `Debug`, so `Result::expect_err` is unavailable.
async fn init_error(params: S3CompatibleWarmBackendParams<'_>, must_fail_because: &str) -> std::io::Error {
match new_s3_compatible_warm_backend(params).await {
Ok(_) => panic!("{must_fail_because}"),
Err(err) => err,
}
}
#[tokio::test]
async fn s3_compatible_backend_rejects_missing_credentials_before_parsing_the_endpoint() {
let mut params = s3_compatible_params("://not-a-url");
params.access_key = "";
let err = init_error(params, "an empty access key must be rejected").await;
assert_eq!(err.to_string(), "both access and secret keys are required");
let mut params = s3_compatible_params("://not-a-url");
params.secret_key = "";
let err = init_error(params, "an empty secret key must be rejected").await;
assert_eq!(err.to_string(), "both access and secret keys are required");
}
#[tokio::test]
async fn s3_compatible_backend_rejects_an_empty_bucket_before_parsing_the_endpoint() {
let mut params = s3_compatible_params("://not-a-url");
params.bucket = "";
let err = init_error(params, "an empty bucket must be rejected").await;
assert_eq!(err.to_string(), "no bucket name was provided");
}
#[tokio::test]
async fn s3_compatible_backend_rejects_an_unparsable_endpoint() {
let err = init_error(s3_compatible_params("://not-a-url"), "an endpoint that is not a URL must be rejected").await;
assert_eq!(err.to_string(), url::ParseError::RelativeUrlWithoutBase.to_string());
}
#[tokio::test]
async fn s3_compatible_backend_rejects_an_endpoint_without_a_host() {
let err = init_error(s3_compatible_params("rustfs://"), "an endpoint without a host must be rejected").await;
assert_eq!(err.to_string(), "Invalid endpoint URL: missing host");
}
/// Every migrated provider that uses `validate_outbound_url` directly (all
/// but RustFS, which injects its own debug-only, env-gated wrapper — see
/// rustfs/rustfs#6773) goes through this one construction path, so the
/// SSRF guard only needs to be pinned here rather than once per provider
/// file (see backlog#2040's migrate steps and rustfs/rustfs#6764).
#[tokio::test]
async fn s3_compatible_backend_rejects_a_loopback_endpoint_before_any_network_setup() {
let err = init_error(s3_compatible_params("https://127.0.0.1:9000"), "a loopback endpoint must be rejected").await;
assert!(err.to_string().contains("not allowed"), "unexpected error: {err}");
}
#[tokio::test]
async fn s3_compatible_backend_carries_provider_options_to_the_transition_client() {
let backend = new_s3_compatible_warm_backend(s3_compatible_params("http://tier.example.com:9000"))
.await
.expect("a well-formed S3-compatible tier config should initialize offline");
assert_eq!(backend.bucket, "tier-bucket");
assert_eq!(backend.prefix, "archive");
assert_eq!(backend.storage_class, "");
assert!(!backend.client.secure);
assert_eq!(backend.client.endpoint_url.scheme(), "http");
assert_eq!(backend.client.endpoint_url.host_str(), Some("tier.example.com"));
assert_eq!(backend.client.endpoint_url.port(), Some(9000));
assert_eq!(backend.client.region, "us-east-1");
assert_eq!(backend.client.lookup, BucketLookupType::BucketLookupDNS);
// The provider constructors all request `trailing_headers: true`, but
// `TransitionClient` gates the feature on an explicitly overridden SigV4
// signer, which none of them set. Pin the resulting `false` so migrating
// a provider onto this constructor cannot silently flip wire behavior.
assert!(!backend.client.trailing_header_support);
assert_eq!(backend.client.tier_type, "aliyun");
}
#[tokio::test]
async fn s3_compatible_backend_derives_tls_and_the_default_port_from_the_scheme() {
let secure = new_s3_compatible_warm_backend(s3_compatible_params("https://tier.example.com"))
.await
.expect("an https endpoint should initialize offline");
assert!(secure.client.secure);
assert_eq!(secure.client.endpoint_url.scheme(), "https");
assert_eq!(secure.client.endpoint_url.port_or_known_default(), Some(443));
let insecure = new_s3_compatible_warm_backend(s3_compatible_params("http://tier.example.com"))
.await
.expect("an http endpoint should initialize offline");
assert!(!insecure.client.secure);
assert_eq!(insecure.client.endpoint_url.scheme(), "http");
assert_eq!(insecure.client.endpoint_url.port_or_known_default(), Some(80));
}
#[tokio::test]
async fn s3_compatible_backend_strips_only_a_trailing_prefix_separator() {
let mut params = s3_compatible_params("http://tier.example.com:9000");
params.prefix = "archive/";
let trimmed = new_s3_compatible_warm_backend(params)
.await
.expect("a prefix with a trailing separator should initialize offline");
assert_eq!(trimmed.prefix, "archive");
let mut params = s3_compatible_params("http://tier.example.com:9000");
params.prefix = "archive/nested";
let untouched = new_s3_compatible_warm_backend(params)
.await
.expect("a nested prefix should initialize offline");
assert_eq!(untouched.prefix, "archive/nested");
let mut params = s3_compatible_params("http://tier.example.com:9000");
params.prefix = "";
let empty = new_s3_compatible_warm_backend(params)
.await
.expect("an empty prefix should initialize offline");
assert_eq!(empty.prefix, "");
}
#[tokio::test]
async fn s3_compatible_backend_honors_the_auto_bucket_lookup_family() {
let mut params = s3_compatible_params("http://tier.example.com:9000");
params.bucket_lookup = BucketLookupType::BucketLookupAuto;
params.provider_tag = "minio";
let backend = new_s3_compatible_warm_backend(params)
.await
.expect("the auto-lookup provider family should initialize offline");
assert_eq!(backend.client.lookup, BucketLookupType::BucketLookupAuto);
assert_eq!(backend.client.tier_type, "minio");
}
#[test]
fn optimal_part_size_charges_an_unknown_length_the_multipart_ceiling() {
let unknown = optimal_part_size(-1, PROVIDER_MIN_PART_SIZE).expect("an unknown length must be accepted");
let ceiling =
optimal_part_size(MAX_MULTIPART_PUT_OBJECT_SIZE, PROVIDER_MIN_PART_SIZE).expect("the exact ceiling must be accepted");
assert_eq!(unknown, ceiling);
assert_eq!(unknown, 5 * PROVIDER_MIN_PART_SIZE);
assert!(unknown * MAX_PARTS_COUNT >= MAX_MULTIPART_PUT_OBJECT_SIZE);
}
#[test]
fn optimal_part_size_rejects_an_object_above_the_multipart_ceiling() {
let err = optimal_part_size(MAX_MULTIPART_PUT_OBJECT_SIZE + 1, PROVIDER_MIN_PART_SIZE)
.expect_err("an object past the multipart ceiling must fail closed");
assert_eq!(err.to_string(), "entity too large");
}
#[test]
fn optimal_part_size_never_returns_less_than_one_part() {
assert_eq!(
optimal_part_size(0, PROVIDER_MIN_PART_SIZE).expect("a zero-length object must be accepted"),
PROVIDER_MIN_PART_SIZE
);
assert_eq!(
optimal_part_size(1024, PROVIDER_MIN_PART_SIZE).expect("a tiny object must be accepted"),
PROVIDER_MIN_PART_SIZE
);
assert_eq!(
optimal_part_size(PROVIDER_MIN_PART_SIZE, PROVIDER_MIN_PART_SIZE)
.expect("an object of exactly one part must be accepted"),
PROVIDER_MIN_PART_SIZE
);
}
#[test]
fn build_transition_put_options_preserves_content_headers() {
let mut metadata = HashMap::new();
@@ -19,76 +19,38 @@
#![allow(clippy::all)]
use std::collections::HashMap;
use std::sync::Arc;
use crate::services::tier::{
tier_config::TierAliyun,
warm_backend::{WarmBackend, WarmBackendGetOpts, build_transition_put_options},
warm_backend::{
S3CompatibleWarmBackendParams, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
new_s3_compatible_warm_backend, optimal_part_size,
},
warm_backend_s3::WarmBackendS3,
};
use rustfs_s3_client::{
admin_handler_utils::AdminError,
api_put_object::PutObjectOptions,
credentials::{Credentials, SignatureType, Static, Value},
transition_api::{BucketLookupType, Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
};
use tracing::warn;
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
use rustfs_utils::egress::validate_outbound_url;
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
const MAX_PARTS_COUNT: i64 = 10000;
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
pub struct WarmBackendAliyun(WarmBackendS3);
impl WarmBackendAliyun {
pub async fn new(conf: &TierAliyun, tier: &str) -> Result<Self, std::io::Error> {
if conf.access_key == "" || conf.secret_key == "" {
return Err(std::io::Error::other("both access and secret keys are required"));
}
if conf.bucket == "" {
return Err(std::io::Error::other("no bucket name was provided"));
}
let u = match url::Url::parse(&conf.endpoint) {
Ok(u) => u,
Err(e) => {
return Err(std::io::Error::other(e.to_string()));
}
};
let creds = Credentials::new(Static(Value {
access_key_id: conf.access_key.clone(),
secret_access_key: conf.secret_key.clone(),
session_token: "".to_string(),
signer_type: SignatureType::SignatureV4,
..Default::default()
}));
let opts = Options {
creds,
secure: u.scheme() == "https",
trailing_headers: true,
region: conf.region.clone(),
bucket_lookup: BucketLookupType::BucketLookupDNS,
..Default::default()
};
let scheme = u.scheme();
let default_port = if scheme == "https" { 443 } else { 80 };
let host = u
.host_str()
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "aliyun").await?;
let client = Arc::new(client);
let core = TransitionCore(Arc::clone(&client));
Ok(Self(WarmBackendS3 {
client,
core,
bucket: conf.bucket.clone(),
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
storage_class: "".to_string(),
}))
Ok(Self(
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
endpoint: &conf.endpoint,
access_key: &conf.access_key,
secret_key: &conf.secret_key,
bucket: &conf.bucket,
prefix: &conf.prefix,
region: &conf.region,
bucket_lookup: BucketLookupType::BucketLookupDNS,
provider_tag: "aliyun",
validate_endpoint: validate_outbound_url,
})
.await?,
))
}
}
@@ -101,7 +63,7 @@ impl WarmBackend for WarmBackendAliyun {
length: i64,
meta: HashMap<String, String>,
) -> Result<String, std::io::Error> {
let part_size = optimal_part_size(length)?;
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
let client = self.0.client.clone();
let res = client
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
@@ -132,23 +94,29 @@ impl WarmBackend for WarmBackendAliyun {
}
}
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
let mut object_size = object_size;
if object_size == -1 {
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
}
#[cfg(test)]
mod tests {
use super::*;
use crate::services::tier::tier_config::TierAliyun;
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
return Err(std::io::Error::other("entity too large"));
}
/// The SSRF guard itself is exercised once, generically, in
/// `warm_backend::tests` (see backlog#2040/backlog#2041 and
/// rustfs/rustfs#6764) — this test only pins that this provider's
/// production constructor really is wired through that shared path.
#[tokio::test]
async fn new_rejects_loopback_endpoint_before_network_setup() {
let conf = TierAliyun {
endpoint: "https://127.0.0.1:9000".to_string(),
bucket: "tier-bucket".to_string(),
access_key: "access".to_string(),
secret_key: "secret".to_string(),
region: "us-east-1".to_string(),
..Default::default()
};
let configured_part_size = MIN_PART_SIZE;
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
let part_size = part_size_flt as i64;
if part_size == 0 {
return Ok(MIN_PART_SIZE);
match WarmBackendAliyun::new(&conf, "tier").await {
Ok(_) => panic!("loopback endpoint should be rejected"),
Err(err) => assert!(err.to_string().contains("not allowed")),
}
}
Ok(part_size)
}
@@ -19,76 +19,38 @@
#![allow(clippy::all)]
use std::collections::HashMap;
use std::sync::Arc;
use crate::services::tier::{
tier_config::TierAzure,
warm_backend::{WarmBackend, WarmBackendGetOpts, build_transition_put_options},
warm_backend::{
S3CompatibleWarmBackendParams, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
new_s3_compatible_warm_backend, optimal_part_size,
},
warm_backend_s3::WarmBackendS3,
};
use rustfs_s3_client::{
admin_handler_utils::AdminError,
api_put_object::PutObjectOptions,
credentials::{Credentials, SignatureType, Static, Value},
transition_api::{BucketLookupType, Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
};
use tracing::warn;
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
use rustfs_utils::egress::validate_outbound_url;
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
const MAX_PARTS_COUNT: i64 = 10000;
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
pub struct WarmBackendAzure(WarmBackendS3);
impl WarmBackendAzure {
pub async fn new(conf: &TierAzure, tier: &str) -> Result<Self, std::io::Error> {
if conf.access_key == "" || conf.secret_key == "" {
return Err(std::io::Error::other("both access and secret keys are required"));
}
if conf.bucket == "" {
return Err(std::io::Error::other("no bucket name was provided"));
}
let u = match url::Url::parse(&conf.endpoint) {
Ok(u) => u,
Err(e) => {
return Err(std::io::Error::other(e.to_string()));
}
};
let creds = Credentials::new(Static(Value {
access_key_id: conf.access_key.clone(),
secret_access_key: conf.secret_key.clone(),
session_token: "".to_string(),
signer_type: SignatureType::SignatureV4,
..Default::default()
}));
let opts = Options {
creds,
secure: u.scheme() == "https",
trailing_headers: true,
region: conf.region.clone(),
bucket_lookup: BucketLookupType::BucketLookupDNS,
..Default::default()
};
let scheme = u.scheme();
let default_port = if scheme == "https" { 443 } else { 80 };
let host = u
.host_str()
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "azure").await?;
let client = Arc::new(client);
let core = TransitionCore(Arc::clone(&client));
Ok(Self(WarmBackendS3 {
client,
core,
bucket: conf.bucket.clone(),
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
storage_class: "".to_string(),
}))
Ok(Self(
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
endpoint: &conf.endpoint,
access_key: &conf.access_key,
secret_key: &conf.secret_key,
bucket: &conf.bucket,
prefix: &conf.prefix,
region: &conf.region,
bucket_lookup: BucketLookupType::BucketLookupDNS,
provider_tag: "azure",
validate_endpoint: validate_outbound_url,
})
.await?,
))
}
}
@@ -101,7 +63,7 @@ impl WarmBackend for WarmBackendAzure {
length: i64,
meta: HashMap<String, String>,
) -> Result<String, std::io::Error> {
let part_size = optimal_part_size(length)?;
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
let client = self.0.client.clone();
let res = client
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
@@ -132,23 +94,29 @@ impl WarmBackend for WarmBackendAzure {
}
}
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
let mut object_size = object_size;
if object_size == -1 {
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
}
#[cfg(test)]
mod tests {
use super::*;
use crate::services::tier::tier_config::TierAzure;
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
return Err(std::io::Error::other("entity too large"));
}
/// The SSRF guard itself is exercised once, generically, in
/// `warm_backend::tests` (see backlog#2040/backlog#2041 and
/// rustfs/rustfs#6764) — this test only pins that this provider's
/// production constructor really is wired through that shared path.
#[tokio::test]
async fn new_rejects_loopback_endpoint_before_network_setup() {
let conf = TierAzure {
endpoint: "https://127.0.0.1:9000".to_string(),
bucket: "tier-bucket".to_string(),
access_key: "access".to_string(),
secret_key: "secret".to_string(),
region: "us-east-1".to_string(),
..Default::default()
};
let configured_part_size = MIN_PART_SIZE;
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
let part_size = part_size_flt as i64;
if part_size == 0 {
return Ok(MIN_PART_SIZE);
match WarmBackendAzure::new(&conf, "tier").await {
Ok(_) => panic!("loopback endpoint should be rejected"),
Err(err) => assert!(err.to_string().contains("not allowed")),
}
}
Ok(part_size)
}
@@ -39,6 +39,7 @@ use rustfs_s3_client::{
api_put_object::PutObjectOptions,
transition_api::{Options, ReadCloser, ReaderImpl},
};
use rustfs_utils::egress::validate_outbound_url;
use tracing::warn;
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
@@ -73,6 +74,12 @@ impl WarmBackendGCS {
return Err(std::io::Error::other("no bucket name was provided"));
}
if !conf.endpoint.is_empty() {
let endpoint_url = url::Url::parse(&conf.endpoint).map_err(|e| std::io::Error::other(e.to_string()))?;
validate_outbound_url(&endpoint_url)
.map_err(|err| std::io::Error::other(format!("tier endpoint is not allowed: {err}")))?;
}
let authorized_user = serde_json::from_str(&conf.creds)?;
let credentials = Builder::new(authorized_user)
//.with_retry_policy(AlwaysRetry.with_attempt_limit(3))
@@ -211,7 +218,9 @@ impl WarmBackend for WarmBackendGCS {
#[cfg(test)]
mod tests {
use super::WarmBackendGCS;
use super::parse_generation;
use crate::services::tier::tier_config::TierGCS;
use std::io::ErrorKind;
#[test]
@@ -231,6 +240,21 @@ mod tests {
assert_eq!(err.kind(), ErrorKind::InvalidData, "{value}");
}
}
#[tokio::test]
async fn new_rejects_loopback_endpoint_before_credential_setup() {
let conf = TierGCS {
endpoint: "https://127.0.0.1:9000".to_string(),
creds: "not-json".to_string(),
bucket: "tier-bucket".to_string(),
..Default::default()
};
match WarmBackendGCS::new(&conf, "tier").await {
Ok(_) => panic!("loopback endpoint should be rejected"),
Err(err) => assert!(err.to_string().contains("not allowed"), "unexpected error: {err}"),
}
}
}
/*fn gcs_to_object_error(err: Error, params: Vec<String>) -> Option<Error> {
@@ -19,77 +19,38 @@
#![allow(clippy::all)]
use std::collections::HashMap;
use std::sync::Arc;
use crate::services::tier::{
tier_config::TierHuaweicloud,
warm_backend::{WarmBackend, WarmBackendGetOpts, build_transition_put_options},
warm_backend::{
S3CompatibleWarmBackendParams, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
new_s3_compatible_warm_backend, optimal_part_size,
},
warm_backend_s3::WarmBackendS3,
};
use rustfs_s3_client::{
admin_handler_utils::AdminError,
api_put_object::PutObjectOptions,
credentials::{Credentials, SignatureType, Static, Value},
transition_api::{BucketLookupType, Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
};
use tracing::warn;
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
use rustfs_utils::egress::validate_outbound_url;
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
const MAX_PARTS_COUNT: i64 = 10000;
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
pub struct WarmBackendHuaweicloud(WarmBackendS3);
impl WarmBackendHuaweicloud {
pub async fn new(conf: &TierHuaweicloud, tier: &str) -> Result<Self, std::io::Error> {
if conf.access_key == "" || conf.secret_key == "" {
return Err(std::io::Error::other("both access and secret keys are required"));
}
if conf.bucket == "" {
return Err(std::io::Error::other("no bucket name was provided"));
}
let u = match url::Url::parse(&conf.endpoint) {
Ok(u) => u,
Err(e) => {
return Err(std::io::Error::other(e.to_string()));
}
};
let creds = Credentials::new(Static(Value {
access_key_id: conf.access_key.clone(),
secret_access_key: conf.secret_key.clone(),
session_token: "".to_string(),
signer_type: SignatureType::SignatureV4,
..Default::default()
}));
let opts = Options {
creds,
secure: u.scheme() == "https",
trailing_headers: true,
region: conf.region.clone(),
bucket_lookup: BucketLookupType::BucketLookupDNS,
..Default::default()
};
let scheme = u.scheme();
let default_port = if scheme == "https" { 443 } else { 80 };
let host = u
.host_str()
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
let client =
TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "huaweicloud").await?;
let client = Arc::new(client);
let core = TransitionCore(Arc::clone(&client));
Ok(Self(WarmBackendS3 {
client,
core,
bucket: conf.bucket.clone(),
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
storage_class: "".to_string(),
}))
Ok(Self(
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
endpoint: &conf.endpoint,
access_key: &conf.access_key,
secret_key: &conf.secret_key,
bucket: &conf.bucket,
prefix: &conf.prefix,
region: &conf.region,
bucket_lookup: BucketLookupType::BucketLookupDNS,
provider_tag: "huaweicloud",
validate_endpoint: validate_outbound_url,
})
.await?,
))
}
}
@@ -102,7 +63,7 @@ impl WarmBackend for WarmBackendHuaweicloud {
length: i64,
meta: HashMap<String, String>,
) -> Result<String, std::io::Error> {
let part_size = optimal_part_size(length)?;
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
let client = self.0.client.clone();
let res = client
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
@@ -133,23 +94,29 @@ impl WarmBackend for WarmBackendHuaweicloud {
}
}
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
let mut object_size = object_size;
if object_size == -1 {
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
}
#[cfg(test)]
mod tests {
use super::*;
use crate::services::tier::tier_config::TierHuaweicloud;
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
return Err(std::io::Error::other("entity too large"));
}
/// The SSRF guard itself is exercised once, generically, in
/// `warm_backend::tests` (see backlog#2040/backlog#2041 and
/// rustfs/rustfs#6764) — this test only pins that this provider's
/// production constructor really is wired through that shared path.
#[tokio::test]
async fn new_rejects_loopback_endpoint_before_network_setup() {
let conf = TierHuaweicloud {
endpoint: "https://127.0.0.1:9000".to_string(),
bucket: "tier-bucket".to_string(),
access_key: "access".to_string(),
secret_key: "secret".to_string(),
region: "us-east-1".to_string(),
..Default::default()
};
let configured_part_size = MIN_PART_SIZE;
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
let part_size = part_size_flt as i64;
if part_size == 0 {
return Ok(MIN_PART_SIZE);
match WarmBackendHuaweicloud::new(&conf, "tier").await {
Ok(_) => panic!("loopback endpoint should be rejected"),
Err(err) => assert!(err.to_string().contains("not allowed")),
}
}
Ok(part_size)
}
@@ -19,23 +19,18 @@
#![allow(clippy::all)]
use std::collections::HashMap;
use std::sync::Arc;
use crate::services::tier::{
tier_config::TierMinIO,
warm_backend::{TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options},
warm_backend::{
S3CompatibleWarmBackendParams, TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
new_s3_compatible_warm_backend, optimal_part_size,
},
warm_backend_s3::WarmBackendS3,
};
use rustfs_s3_client::{
admin_handler_utils::AdminError,
api_put_object::PutObjectOptions,
credentials::{Credentials, SignatureType, Static, Value},
transition_api::{Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
};
use tracing::warn;
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
use rustfs_utils::egress::validate_outbound_url;
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
const MAX_PARTS_COUNT: i64 = 10000;
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
@@ -43,51 +38,22 @@ pub struct WarmBackendMinIO(WarmBackendS3);
impl WarmBackendMinIO {
pub async fn new(conf: &TierMinIO, tier: &str) -> Result<Self, std::io::Error> {
if conf.access_key == "" || conf.secret_key == "" {
return Err(std::io::Error::other("both access and secret keys are required"));
}
if conf.bucket == "" {
return Err(std::io::Error::other("no bucket name was provided"));
}
let u = match url::Url::parse(&conf.endpoint) {
Ok(u) => u,
Err(e) => {
return Err(std::io::Error::other(e.to_string()));
}
};
let creds = Credentials::new(Static(Value {
access_key_id: conf.access_key.clone(),
secret_access_key: conf.secret_key.clone(),
session_token: "".to_string(),
signer_type: SignatureType::SignatureV4,
..Default::default()
}));
let opts = Options {
creds,
secure: u.scheme() == "https",
trailing_headers: true,
region: conf.region.clone(),
..Default::default()
};
let scheme = u.scheme();
let default_port = if scheme == "https" { 443 } else { 80 };
let host = u
.host_str()
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "minio").await?;
let client = Arc::new(client);
let core = TransitionCore(Arc::clone(&client));
Ok(Self(WarmBackendS3 {
client,
core,
bucket: conf.bucket.clone(),
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
storage_class: "".to_string(),
}))
Ok(Self(
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
endpoint: &conf.endpoint,
access_key: &conf.access_key,
secret_key: &conf.secret_key,
bucket: &conf.bucket,
prefix: &conf.prefix,
region: &conf.region,
// MinIO tier endpoints are commonly path-style, so bucket addressing stays on
// `BucketLookupAuto`; pinning DNS here would break those deployments.
bucket_lookup: BucketLookupType::BucketLookupAuto,
provider_tag: "minio",
validate_endpoint: validate_outbound_url,
})
.await?,
))
}
}
@@ -100,7 +66,7 @@ impl WarmBackend for WarmBackendMinIO {
length: i64,
meta: HashMap<String, String>,
) -> Result<String, std::io::Error> {
let part_size = optimal_part_size(length)?;
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
let client = self.0.client.clone();
let res = client
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
@@ -149,23 +115,29 @@ impl crate::services::tier::warm_backend::TransitionCandidateReconciler for Warm
}
}
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
let mut object_size = object_size;
if object_size == -1 {
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
}
#[cfg(test)]
mod tests {
use super::*;
use crate::services::tier::tier_config::TierMinIO;
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
return Err(std::io::Error::other("entity too large"));
}
/// The SSRF guard itself is exercised once, generically, in
/// `warm_backend::tests` (see backlog#2040/backlog#2043 and
/// rustfs/rustfs#6764) — this test only pins that this provider's
/// production constructor really is wired through that shared path.
#[tokio::test]
async fn new_rejects_loopback_endpoint_before_network_setup() {
let conf = TierMinIO {
endpoint: "https://127.0.0.1:9000".to_string(),
bucket: "tier-bucket".to_string(),
access_key: "access".to_string(),
secret_key: "secret".to_string(),
region: "us-east-1".to_string(),
..Default::default()
};
let configured_part_size = MIN_PART_SIZE;
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
let part_size = part_size_flt as i64;
if part_size == 0 {
return Ok(MIN_PART_SIZE);
match WarmBackendMinIO::new(&conf, "tier").await {
Ok(_) => panic!("loopback endpoint should be rejected"),
Err(err) => assert!(err.to_string().contains("not allowed")),
}
}
Ok(part_size)
}
@@ -19,23 +19,18 @@
#![allow(clippy::all)]
use std::collections::HashMap;
use std::sync::Arc;
use crate::services::tier::{
tier_config::TierR2,
warm_backend::{TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options},
warm_backend::{
S3CompatibleWarmBackendParams, TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
new_s3_compatible_warm_backend, optimal_part_size,
},
warm_backend_s3::WarmBackendS3,
};
use rustfs_s3_client::{
admin_handler_utils::AdminError,
api_put_object::PutObjectOptions,
credentials::{Credentials, SignatureType, Static, Value},
transition_api::{Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
};
use tracing::warn;
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
use rustfs_utils::egress::validate_outbound_url;
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
const MAX_PARTS_COUNT: i64 = 10000;
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
@@ -43,51 +38,22 @@ pub struct WarmBackendR2(WarmBackendS3);
impl WarmBackendR2 {
pub async fn new(conf: &TierR2, tier: &str) -> Result<Self, std::io::Error> {
if conf.access_key == "" || conf.secret_key == "" {
return Err(std::io::Error::other("both access and secret keys are required"));
}
if conf.bucket == "" {
return Err(std::io::Error::other("no bucket name was provided"));
}
let u = match url::Url::parse(&conf.endpoint) {
Ok(u) => u,
Err(e) => {
return Err(std::io::Error::other(e.to_string()));
}
};
let creds = Credentials::new(Static(Value {
access_key_id: conf.access_key.clone(),
secret_access_key: conf.secret_key.clone(),
session_token: "".to_string(),
signer_type: SignatureType::SignatureV4,
..Default::default()
}));
let opts = Options {
creds,
secure: u.scheme() == "https",
trailing_headers: true,
region: conf.region.clone(),
..Default::default()
};
let scheme = u.scheme();
let default_port = if scheme == "https" { 443 } else { 80 };
let host = u
.host_str()
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "r2").await?;
let client = Arc::new(client);
let core = TransitionCore(Arc::clone(&client));
Ok(Self(WarmBackendS3 {
client,
core,
bucket: conf.bucket.clone(),
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
storage_class: "".to_string(),
}))
Ok(Self(
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
endpoint: &conf.endpoint,
access_key: &conf.access_key,
secret_key: &conf.secret_key,
bucket: &conf.bucket,
prefix: &conf.prefix,
region: &conf.region,
// R2 tier endpoints are commonly path-style, so bucket addressing stays on
// `BucketLookupAuto`; pinning DNS here would break those deployments.
bucket_lookup: BucketLookupType::BucketLookupAuto,
provider_tag: "r2",
validate_endpoint: validate_outbound_url,
})
.await?,
))
}
}
@@ -100,7 +66,7 @@ impl WarmBackend for WarmBackendR2 {
length: i64,
meta: HashMap<String, String>,
) -> Result<String, std::io::Error> {
let part_size = optimal_part_size(length)?;
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
let client = self.0.client.clone();
let res = client
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
@@ -149,23 +115,29 @@ impl crate::services::tier::warm_backend::TransitionCandidateReconciler for Warm
}
}
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
let mut object_size = object_size;
if object_size == -1 {
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
}
#[cfg(test)]
mod tests {
use super::*;
use crate::services::tier::tier_config::TierR2;
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
return Err(std::io::Error::other("entity too large"));
}
/// The SSRF guard itself is exercised once, generically, in
/// `warm_backend::tests` (see backlog#2040/backlog#2043 and
/// rustfs/rustfs#6764) — this test only pins that this provider's
/// production constructor really is wired through that shared path.
#[tokio::test]
async fn new_rejects_loopback_endpoint_before_network_setup() {
let conf = TierR2 {
endpoint: "https://127.0.0.1:9000".to_string(),
bucket: "tier-bucket".to_string(),
access_key: "access".to_string(),
secret_key: "secret".to_string(),
region: "us-east-1".to_string(),
..Default::default()
};
let configured_part_size = MIN_PART_SIZE;
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
let part_size = part_size_flt as i64;
if part_size == 0 {
return Ok(MIN_PART_SIZE);
match WarmBackendR2::new(&conf, "tier").await {
Ok(_) => panic!("loopback endpoint should be rejected"),
Err(err) => assert!(err.to_string().contains("not allowed")),
}
}
Ok(part_size)
}
@@ -19,34 +19,55 @@
#![allow(clippy::all)]
use std::collections::HashMap;
use std::sync::Arc;
use crate::services::tier::{
tier_config::TierRustFS,
warm_backend::{TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options},
warm_backend::{
S3CompatibleWarmBackendParams, TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
new_s3_compatible_warm_backend, optimal_part_size,
},
warm_backend_s3::WarmBackendS3,
};
use rustfs_s3_client::{
admin_handler_utils::AdminError,
api_put_object::PutObjectOptions,
credentials::{Credentials, SignatureType, Static, Value},
transition_api::{Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
};
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
use rustfs_utils::egress::{OutboundUrlError, validate_outbound_url};
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
const MAX_PARTS_COUNT: i64 = 10000;
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
// Debug-only opt-in for single-host test/dev setups; release builds always reject loopback.
const ALLOW_LOOPBACK_TIER_ENDPOINT_ENV: &str = "RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT";
fn validate_rustfs_tier_endpoint(url: &url::Url) -> Result<(), OutboundUrlError> {
let allow_loopback = cfg!(debug_assertions)
&& std::env::var(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV)
.map(|value| value == "1" || value.eq_ignore_ascii_case("true"))
.unwrap_or(false);
validate_rustfs_tier_endpoint_inner(url, allow_loopback)
}
fn validate_rustfs_tier_endpoint_inner(url: &url::Url, allow_loopback: bool) -> Result<(), OutboundUrlError> {
match validate_outbound_url(url) {
Err(OutboundUrlError::ForbiddenHost {
reason: "loopback address" | "loopback host",
..
}) if allow_loopback => Ok(()),
result => result,
}
}
pub struct WarmBackendRustFS(WarmBackendS3);
impl WarmBackendRustFS {
pub async fn new(conf: &TierRustFS, tier: &str) -> Result<Self, std::io::Error> {
if conf.access_key == "" || conf.secret_key == "" {
// This provider reports endpoint problems with its own wording (and keeps the
// `url::ParseError` as the io::Error source) while the shared constructor carries the
// MinIO-derived texts. Unifying the two is a separate change, so the endpoint is
// pre-validated here, after the credential and bucket checks so the order in which the
// shared constructor would report the same failures is preserved.
if conf.access_key.is_empty() || conf.secret_key.is_empty() {
return Err(std::io::Error::other("both access and secret keys are required"));
}
if conf.bucket == "" {
if conf.bucket.is_empty() {
return Err(std::io::Error::other("no bucket name was provided"));
}
@@ -55,36 +76,29 @@ impl WarmBackendRustFS {
Err(e) => return Err(std::io::Error::other(e)),
};
let creds = Credentials::new(Static(Value {
access_key_id: conf.access_key.clone(),
secret_access_key: conf.secret_key.clone(),
session_token: "".to_string(),
signer_type: SignatureType::SignatureV4,
..Default::default()
}));
let opts = Options {
creds,
secure: u.scheme() == "https",
trailing_headers: true,
region: conf.region.clone(),
..Default::default()
};
let scheme = u.scheme();
let default_port = if scheme == "https" { 443 } else { 80 };
let host = u
.host_str()
.ok_or_else(|| std::io::Error::other("endpoint URL must include a host"))?;
let client = TransitionClient::new(&format!("{host}:{}", u.port().unwrap_or(default_port)), opts, "rustfs").await?;
if u.host_str().is_none() {
return Err(std::io::Error::other("endpoint URL must include a host"));
}
let client = Arc::new(client);
let core = TransitionCore(Arc::clone(&client));
Ok(Self(WarmBackendS3 {
client,
core,
bucket: conf.bucket.clone(),
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
storage_class: "".to_string(),
}))
Ok(Self(
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
endpoint: &conf.endpoint,
access_key: &conf.access_key,
secret_key: &conf.secret_key,
bucket: &conf.bucket,
prefix: &conf.prefix,
region: &conf.region,
// RustFS tier endpoints are path-style, so bucket addressing stays on
// `BucketLookupAuto`; pinning DNS here would break those endpoints.
bucket_lookup: BucketLookupType::BucketLookupAuto,
provider_tag: "rustfs",
// Debug-only, env-gated loopback exception for this provider's own e2e tier
// tests (rustfs/rustfs#6773); every other provider passes plain
// `validate_outbound_url`.
validate_endpoint: validate_rustfs_tier_endpoint,
})
.await?,
))
}
}
@@ -97,7 +111,7 @@ impl WarmBackend for WarmBackendRustFS {
length: i64,
meta: HashMap<String, String>,
) -> Result<String, std::io::Error> {
let part_size = optimal_part_size(length)?;
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
let client = self.0.client.clone();
let res = client
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
@@ -146,27 +160,6 @@ impl crate::services::tier::warm_backend::TransitionCandidateReconciler for Warm
}
}
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
let mut object_size = object_size;
if object_size == -1 {
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
}
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
return Err(std::io::Error::other("entity too large"));
}
let configured_part_size = MIN_PART_SIZE;
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
let part_size = part_size_flt as i64;
if part_size == 0 {
return Ok(MIN_PART_SIZE);
}
Ok(part_size)
}
#[cfg(test)]
mod tests {
use futures::FutureExt;
@@ -197,4 +190,23 @@ mod tests {
};
assert!(err.to_string().contains("host"), "expected host validation error, got: {err}");
}
#[tokio::test]
async fn new_rejects_loopback_endpoint_before_network_setup() {
let conf = rustfs_tier("https://127.0.0.1:9000");
match WarmBackendRustFS::new(&conf, "tier").await {
Ok(_) => panic!("loopback endpoint should be rejected"),
Err(err) => assert!(err.to_string().contains("not allowed")),
}
}
#[test]
fn loopback_opt_in_does_not_allow_other_private_endpoints() {
let loopback = url::Url::parse("https://127.0.0.1:9000").unwrap();
assert!(validate_rustfs_tier_endpoint_inner(&loopback, true).is_ok());
let private = url::Url::parse("https://10.0.0.1:9000").unwrap();
assert!(validate_rustfs_tier_endpoint_inner(&private, true).is_err());
}
}
@@ -19,76 +19,38 @@
#![allow(clippy::all)]
use std::collections::HashMap;
use std::sync::Arc;
use crate::services::tier::{
tier_config::TierTencent,
warm_backend::{WarmBackend, WarmBackendGetOpts, build_transition_put_options},
warm_backend::{
S3CompatibleWarmBackendParams, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
new_s3_compatible_warm_backend, optimal_part_size,
},
warm_backend_s3::WarmBackendS3,
};
use rustfs_s3_client::{
admin_handler_utils::AdminError,
api_put_object::PutObjectOptions,
credentials::{Credentials, SignatureType, Static, Value},
transition_api::{BucketLookupType, Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
};
use tracing::warn;
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
use rustfs_utils::egress::validate_outbound_url;
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
const MAX_PARTS_COUNT: i64 = 10000;
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
pub struct WarmBackendTencent(WarmBackendS3);
impl WarmBackendTencent {
pub async fn new(conf: &TierTencent, tier: &str) -> Result<Self, std::io::Error> {
if conf.access_key == "" || conf.secret_key == "" {
return Err(std::io::Error::other("both access and secret keys are required"));
}
if conf.bucket == "" {
return Err(std::io::Error::other("no bucket name was provided"));
}
let u = match url::Url::parse(&conf.endpoint) {
Ok(u) => u,
Err(e) => {
return Err(std::io::Error::other(e.to_string()));
}
};
let creds = Credentials::new(Static(Value {
access_key_id: conf.access_key.clone(),
secret_access_key: conf.secret_key.clone(),
session_token: "".to_string(),
signer_type: SignatureType::SignatureV4,
..Default::default()
}));
let opts = Options {
creds,
secure: u.scheme() == "https",
trailing_headers: true,
region: conf.region.clone(),
bucket_lookup: BucketLookupType::BucketLookupDNS,
..Default::default()
};
let scheme = u.scheme();
let default_port = if scheme == "https" { 443 } else { 80 };
let host = u
.host_str()
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "tencent").await?;
let client = Arc::new(client);
let core = TransitionCore(Arc::clone(&client));
Ok(Self(WarmBackendS3 {
client,
core,
bucket: conf.bucket.clone(),
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
storage_class: "".to_string(),
}))
Ok(Self(
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
endpoint: &conf.endpoint,
access_key: &conf.access_key,
secret_key: &conf.secret_key,
bucket: &conf.bucket,
prefix: &conf.prefix,
region: &conf.region,
bucket_lookup: BucketLookupType::BucketLookupDNS,
provider_tag: "tencent",
validate_endpoint: validate_outbound_url,
})
.await?,
))
}
}
@@ -101,7 +63,7 @@ impl WarmBackend for WarmBackendTencent {
length: i64,
meta: HashMap<String, String>,
) -> Result<String, std::io::Error> {
let part_size = optimal_part_size(length)?;
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
let client = self.0.client.clone();
let res = client
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
@@ -132,23 +94,29 @@ impl WarmBackend for WarmBackendTencent {
}
}
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
let mut object_size = object_size;
if object_size == -1 {
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
}
#[cfg(test)]
mod tests {
use super::*;
use crate::services::tier::tier_config::TierTencent;
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
return Err(std::io::Error::other("entity too large"));
}
/// The SSRF guard itself is exercised once, generically, in
/// `warm_backend::tests` (see backlog#2040/backlog#2041 and
/// rustfs/rustfs#6764) — this test only pins that this provider's
/// production constructor really is wired through that shared path.
#[tokio::test]
async fn new_rejects_loopback_endpoint_before_network_setup() {
let conf = TierTencent {
endpoint: "https://127.0.0.1:9000".to_string(),
bucket: "tier-bucket".to_string(),
access_key: "access".to_string(),
secret_key: "secret".to_string(),
region: "us-east-1".to_string(),
..Default::default()
};
let configured_part_size = MIN_PART_SIZE;
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
let part_size = part_size_flt as i64;
if part_size == 0 {
return Ok(MIN_PART_SIZE);
match WarmBackendTencent::new(&conf, "tier").await {
Ok(_) => panic!("loopback endpoint should be rejected"),
Err(err) => assert!(err.to_string().contains("not allowed")),
}
}
Ok(part_size)
}
@@ -5696,6 +5696,8 @@ impl SetDisks {
{
let grace = dangling_delete_grace();
if !grace.is_zero() && OffsetDateTime::now_utc() - mod_time < grace {
let elapsed = OffsetDateTime::now_utc() - mod_time;
let retry_after_secs = grace.saturating_sub(elapsed).whole_seconds().max(0);
info!(
bucket = bucket,
object = object,
@@ -5703,7 +5705,7 @@ impl SetDisks {
grace_secs = grace.whole_seconds(),
"skipping dangling-object deletion within grace window"
);
return Err(DiskError::ErasureReadQuorum);
return Err(DiskError::dangling_delete_grace(retry_after_secs, grace.whole_seconds()));
}
}
@@ -6799,6 +6801,7 @@ pub(in crate::set_disk) mod rename_fanout_barrier {
#[cfg(test)]
mod tests {
use crate::disk::error::HEAL_DANGLING_DELETE_GRACE_MESSAGE;
use crate::disk::local::{DurabilityMode, durability_mode_override};
use super::*;
@@ -10746,7 +10749,13 @@ mod tests {
let object = "object";
let (_dir, disk) = read_multiple_test_disk(bucket, &[]).await;
let set = io_primitives_test_set(vec![Some(disk.clone()), None, None], 1).await;
let mut fi = metadata_test_fileinfo(object);
let mut fi = FileInfo::new(object, 2, 1);
fi.volume = bucket.to_string();
fi.name = object.to_string();
fi.size = 1;
fi.erasure.index = 1;
fi.metadata.insert("etag".to_string(), "etag-1".to_string());
fi.add_object_part(1, "part-etag-1".to_string(), 1, None, 1, None, None);
fi.mod_time = Some(OffsetDateTime::now_utc());
disk.write_metadata(bucket, bucket, object, fi.clone())
.await
@@ -10764,7 +10773,15 @@ mod tests {
.await
.expect_err("recent dangling metadata must stay protected by grace");
assert_eq!(err, DiskError::ErasureReadQuorum);
let message = err.to_string();
assert!(
message.contains(HEAL_DANGLING_DELETE_GRACE_MESSAGE),
"grace-protected dangling cleanup must explain the deferred delete: {message}"
);
assert!(
message.contains("retry_after_secs="),
"grace-protected dangling cleanup must include retry timing: {message}"
);
disk.read_all(bucket, &path_join_buf(&[object, STORAGE_FORMAT_FILE]))
.await
.expect("metadata should remain during dangling grace");
+39 -10
View File
@@ -575,18 +575,12 @@ impl SetDisks {
meta.metadata.keys().any(|name| http::is_object_encryption_marker(name))
}
fn starts_with_ignore_ascii_case(value: &str, prefix: &str) -> bool {
value
.get(..prefix.len())
.is_some_and(|value_prefix| value_prefix.eq_ignore_ascii_case(prefix))
}
fn internal_metadata_suffix(name: &str) -> Option<&str> {
name.get(http::RUSTFS_INTERNAL_PREFIX.len()..)
.filter(|_| Self::starts_with_ignore_ascii_case(name, http::RUSTFS_INTERNAL_PREFIX))
.filter(|_| http::starts_with_ignore_ascii_case(name, http::RUSTFS_INTERNAL_PREFIX))
.or_else(|| {
name.get(http::MINIO_INTERNAL_PREFIX.len()..)
.filter(|_| Self::starts_with_ignore_ascii_case(name, http::MINIO_INTERNAL_PREFIX))
.filter(|_| http::starts_with_ignore_ascii_case(name, http::MINIO_INTERNAL_PREFIX))
})
}
@@ -604,9 +598,9 @@ impl SetDisks {
|| suffix.eq_ignore_ascii_case(http::SUFFIX_REPLICATION_STATUS)
|| suffix.eq_ignore_ascii_case(http::SUFFIX_REPLICATION_TIMESTAMP)
|| suffix.eq_ignore_ascii_case(http::SUFFIX_PURGESTATUS)
|| Self::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_RESET_ARN_PREFIX)
|| http::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_RESET_ARN_PREFIX)
// Raw compatibility keys are normalized and hashed separately below.
|| Self::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX)
|| http::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX)
}
fn update_hash_quorum_metadata_map(hasher: &mut Sha256, entries: &HashMap<String, String>) {
@@ -1590,6 +1584,41 @@ mod tests {
);
}
/// Guards the switch to `rustfs_utils::http::starts_with_ignore_ascii_case`:
/// internal prefixes must keep matching case-insensitively, and keys shorter
/// than the prefix must keep being rejected. Misclassifying either way leaks
/// internal metadata into the quorum hash (or drops it out of it).
#[test]
fn internal_metadata_suffix_is_prefix_case_insensitive_and_rejects_short_keys() {
assert_eq!(
SetDisks::internal_metadata_suffix("X-RustFS-Internal-Replica-Status"),
Some("Replica-Status"),
"mixed-case RustFS prefix must match and preserve the suffix casing"
);
assert_eq!(
SetDisks::internal_metadata_suffix("X-MINIO-INTERNAL-replica-status"),
Some("replica-status"),
"mixed-case MinIO prefix must match"
);
assert_eq!(SetDisks::internal_metadata_suffix(http::RUSTFS_INTERNAL_PREFIX), Some(""));
// Keys shorter than either prefix, and non-internal keys, stay unmatched.
assert_eq!(SetDisks::internal_metadata_suffix(""), None);
assert_eq!(SetDisks::internal_metadata_suffix("x-rustfs-interna"), None);
assert_eq!(SetDisks::internal_metadata_suffix("x-minio-interna"), None);
assert_eq!(SetDisks::internal_metadata_suffix("x-amz-meta-custom"), None);
// The suffix-prefix comparisons behind the classifier follow the same rules.
assert!(SetDisks::is_replication_quorum_metadata_key(
"X-RustFS-Internal-Replication-Reset-arn:rustfs:replication::target:bucket"
));
assert!(SetDisks::is_replication_quorum_metadata_key(
"X-Minio-Internal-Replication-Delete-Marker-Version-arn:rustfs:replication::target:bucket"
));
assert!(!SetDisks::is_replication_quorum_metadata_key("x-rustfs-interna"));
assert!(!SetDisks::is_replication_quorum_metadata_key("x-rustfs-internal-replication-res"));
}
/// rustfs#5801: parity counts outside [0, total_shards] come from corrupt
/// or foreign metadata and must be treated as invalid entries instead of
/// clamped values that poison `common_parity`'s occurrence counting.
+433 -5
View File
@@ -740,6 +740,10 @@ const ENV_RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY: &str = "RUSTFS_GET_SMALL_OBJECT
const DEFAULT_RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY: bool = true;
const ENV_RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY_THRESHOLD: &str = "RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY_THRESHOLD";
const DEFAULT_RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY_THRESHOLD: usize = 128 * 1024;
// Bound the in-memory inline decoder independently from the configurable
// storage-class admission policy. This protects its Vec allocation while
// allowing explicitly configured EC layouts to inline objects above 128 KiB.
const INLINE_FAST_PATH_MAX_OBJECT_SIZE: i64 = 1024 * 1024;
// --- Metadata Early-Stop Configuration ---
@@ -2285,6 +2289,14 @@ fn get_codec_streaming_config() -> GetCodecStreamingConfig {
}
}
/// Shared kill-switch and compatibility guard for all non-duplex GET readers.
/// Size-specific readers may have independent rollout gates, but they must not
/// bypass these emergency controls.
pub(super) fn is_get_codec_streaming_base_enabled() -> bool {
let config = get_codec_streaming_config();
config.enabled && config.body_compat_confirmed && config.header_compat_confirmed
}
fn get_codec_streaming_engine() -> GetCodecStreamingEngine {
get_codec_streaming_config().engine
}
@@ -2345,6 +2357,7 @@ enum GetCodecStreamingFallbackReason {
ReadQuorumNotSafe,
MultipartPartLimit,
CopySourceDemandBound,
InvalidMetadataShape,
}
impl GetCodecStreamingFallbackReason {
@@ -2367,6 +2380,7 @@ impl GetCodecStreamingFallbackReason {
Self::ReadQuorumNotSafe => "read_quorum_not_safe",
Self::MultipartPartLimit => "multipart_part_limit",
Self::CopySourceDemandBound => "copy_source_demand_bound",
Self::InvalidMetadataShape => "invalid_metadata_shape",
}
}
}
@@ -2426,6 +2440,7 @@ enum GetDirectMemoryFallbackReason {
Remote,
ObjectInfoMultipart,
FileInfoMultipart,
MetadataShape,
InvalidSize,
SizeMismatch,
AboveThreshold,
@@ -2451,6 +2466,7 @@ impl GetDirectMemoryFallbackReason {
Self::Remote => "remote",
Self::ObjectInfoMultipart => "object_info_multipart",
Self::FileInfoMultipart => "file_info_multipart",
Self::MetadataShape => "metadata_shape",
Self::InvalidSize => "invalid_size",
Self::SizeMismatch => "size_mismatch",
Self::AboveThreshold => "above_threshold",
@@ -2499,10 +2515,42 @@ fn record_get_object_reader_path_observation(
object_class: GetCodecStreamingObjectClass,
size_bucket: &'static str,
) {
#[cfg(test)]
LAST_GET_OBJECT_READER_PATH.store(
match path {
crate::set_disk::read::GET_OBJECT_PATH_MID_SIZE_STREAMING => 1,
GET_OBJECT_PATH_DIRECT_MEMORY => 2,
GET_OBJECT_PATH_INLINE_DIRECT => 3,
GET_OBJECT_PATH_BODY_CACHE => 4,
GET_OBJECT_PATH_CODEC_STREAMING => 5,
GET_OBJECT_PATH_REMOTE_TRANSITION => 6,
GET_OBJECT_PATH_EMPTY => 7,
_ => 255,
},
Ordering::Relaxed,
);
rustfs_io_metrics::record_get_object_reader_path(path);
rustfs_io_metrics::record_get_object_reader_path_by_size(path, object_class.as_str(), size_bucket);
}
#[cfg(test)]
static LAST_GET_OBJECT_READER_PATH: AtomicU64 = AtomicU64::new(0);
#[cfg(test)]
pub(crate) fn reset_test_get_object_reader_path() {
LAST_GET_OBJECT_READER_PATH.store(0, Ordering::Relaxed);
}
#[cfg(test)]
pub(crate) fn test_get_object_reader_selected_mid_size() -> bool {
LAST_GET_OBJECT_READER_PATH.load(Ordering::Relaxed) == 1
}
#[cfg(test)]
pub(crate) fn test_get_object_reader_path_id() -> u64 {
LAST_GET_OBJECT_READER_PATH.load(Ordering::Relaxed)
}
fn classify_get_codec_streaming_object_class(
range: &Option<HTTPRangeSpec>,
object_info: &ObjectInfo,
@@ -2547,6 +2595,26 @@ fn get_small_object_direct_memory_decision_with_threshold(
opts: &ObjectOptions,
enabled: bool,
threshold: usize,
) -> GetDirectMemoryDecision {
get_small_object_direct_memory_decision_with_threshold_and_plan(
range,
object_info,
fi,
opts,
enabled,
threshold,
ReadPathPlan::new(object_info, fi),
)
}
fn get_small_object_direct_memory_decision_with_threshold_and_plan(
range: &Option<HTTPRangeSpec>,
object_info: &ObjectInfo,
fi: &FileInfo,
opts: &ObjectOptions,
enabled: bool,
threshold: usize,
plan: ReadPathPlan,
) -> GetDirectMemoryDecision {
if !enabled {
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::Disabled);
@@ -2605,14 +2673,24 @@ fn get_small_object_direct_memory_decision_with_threshold(
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::FileInfoMultipart);
}
let Ok(object_size) = usize::try_from(fi.size) else {
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::InvalidSize);
if object_info.size != fi.size {
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::SizeMismatch);
}
let Some(shape) = plan.shape() else {
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::MetadataShape);
};
let object_size = shape.object_size;
if object_size == 0 {
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::InvalidSize);
}
if object_info.size != fi.size {
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::SizeMismatch);
if !plan.is_plain() {
if object_info.is_encrypted() || fi.metadata.keys().any(|key| is_object_encryption_marker(key)) {
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::Encrypted);
}
if object_info.is_compressed() || fi.is_compressed() {
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::Compressed);
}
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::Remote);
}
if object_size > threshold {
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::AboveThreshold);
@@ -2621,6 +2699,7 @@ fn get_small_object_direct_memory_decision_with_threshold(
GetDirectMemoryDecision::Use { object_size }
}
#[allow(dead_code, reason = "asserted by this file's gate tests")]
fn get_small_object_direct_memory_decision(
range: &Option<HTTPRangeSpec>,
object_info: &ObjectInfo,
@@ -2655,6 +2734,7 @@ fn should_prefer_codec_streaming_data_blocks_first_reader_setup(
max_size > 0 && object_size <= max_size
}
#[allow(dead_code, reason = "asserted by this file's gate tests")]
fn get_codec_streaming_reader_gate(
bucket: &str,
object: &str,
@@ -2663,6 +2743,29 @@ fn get_codec_streaming_reader_gate(
object_info: &ObjectInfo,
fi: &FileInfo,
lock_optimization_enabled: bool,
) -> GetCodecStreamingGate {
get_codec_streaming_reader_gate_with_plan(
bucket,
object,
part_number,
object_class,
object_info,
fi,
lock_optimization_enabled,
ReadPathPlan::new(object_info, fi),
)
}
#[allow(clippy::too_many_arguments, reason = "keeps the hot-path gate inputs explicit")]
fn get_codec_streaming_reader_gate_with_plan(
bucket: &str,
object: &str,
part_number: Option<usize>,
object_class: GetCodecStreamingObjectClass,
object_info: &ObjectInfo,
fi: &FileInfo,
lock_optimization_enabled: bool,
plan: ReadPathPlan,
) -> GetCodecStreamingGate {
let config = get_codec_streaming_config();
@@ -2778,6 +2881,22 @@ fn get_codec_streaming_reader_gate(
};
}
}
if object_class == GetCodecStreamingObjectClass::PlainSinglePart {
let Some(_shape) = plan.shape() else {
return GetCodecStreamingGate {
object_class,
decision: GetCodecStreamingDecision::Fallback(GetCodecStreamingFallbackReason::InvalidMetadataShape),
prefer_data_blocks_first_reader_setup: false,
};
};
if !plan.is_plain() {
return GetCodecStreamingGate {
object_class,
decision: GetCodecStreamingDecision::Fallback(GetCodecStreamingFallbackReason::InvalidMetadataShape),
prefer_data_blocks_first_reader_setup: false,
};
}
}
let Ok(min_size) = i64::try_from(config.min_size) else {
return GetCodecStreamingGate {
object_class,
@@ -4062,6 +4181,80 @@ fn object_fits_single_block(object_size: i64, block_size: usize) -> bool {
}
}
/// The common metadata contract consumed by all bounded GET fast paths.
///
/// `ObjectInfo` is assembled from `FileInfo`, but callers may also provide a
/// prepared snapshot or metadata from an older peer. Never let either copy
/// independently decide that a request is safe: a disagreement must fall
/// back to the regular reader. The returned size is the only size used for
/// fast-path allocation and reader setup.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(super) struct ReadPathShape {
pub(super) object_size: usize,
}
impl ReadPathShape {
pub(super) fn is_plain(self, object_info: &ObjectInfo, fi: &FileInfo) -> bool {
!object_info.is_encrypted()
&& !fi.metadata.keys().any(|key| is_object_encryption_marker(key))
&& !object_info.is_compressed()
&& !fi.is_compressed()
&& !object_info.is_remote()
&& !fi.is_remote()
}
}
/// Request-local metadata decision reused by all small-object GET gates.
///
/// The metadata pair is immutable for the lifetime of `get_object_reader`, so
/// validating it once avoids repeating part-array and transform scans on every
/// fast-path predicate while keeping the trust boundary fail-closed.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
struct ReadPathPlan {
shape: Option<ReadPathShape>,
plain: bool,
}
impl ReadPathPlan {
fn new(object_info: &ObjectInfo, fi: &FileInfo) -> Self {
let shape = read_path_shape(object_info, fi);
let plain = shape.is_some_and(|shape| shape.is_plain(object_info, fi));
Self { shape, plain }
}
const fn shape(self) -> Option<ReadPathShape> {
self.shape
}
const fn is_plain(self) -> bool {
self.plain
}
}
/// Validate the single-part geometry shared by inline, direct-memory, and
/// bounded mid-size readers. This is deliberately fail-closed: a stale or
/// mixed-version `ObjectInfo`/`FileInfo` pair must use the legacy path rather
/// than risk allocating or decoding with a mismatched size.
pub(super) fn read_path_shape(object_info: &ObjectInfo, fi: &FileInfo) -> Option<ReadPathShape> {
if object_info.parts.len() != 1 || fi.parts.len() != 1 || object_info.size < 0 || fi.size < 0 || object_info.size != fi.size {
return None;
}
let object_part = object_info.parts.first()?;
let file_part = fi.parts.first()?;
let object_size = usize::try_from(fi.size).ok()?;
if object_part.number != file_part.number
|| object_part.size != file_part.size
|| object_part.actual_size != file_part.actual_size
|| file_part.size != object_size
|| file_part.actual_size != fi.size
{
return None;
}
Some(ReadPathShape { object_size })
}
fn should_use_inline_small_fast_path(is_inline_buffer: bool, object_size: i64, block_size: usize) -> bool {
is_inline_buffer && object_fits_single_block(object_size, block_size)
}
@@ -4070,13 +4263,43 @@ fn should_use_single_block_non_inline_fast_path(is_inline_buffer: bool, object_s
!is_inline_buffer && object_fits_single_block(object_size, block_size)
}
#[allow(dead_code, reason = "asserted by this file's gate tests")]
fn should_use_inline_fast_path(
range: &Option<HTTPRangeSpec>,
object_info: &ObjectInfo,
fi: &FileInfo,
opts: &ObjectOptions,
) -> bool {
object_info.is_inline_fast_path_eligible() && fi.data.is_some() && range.is_none() && opts.part_number.is_none()
should_use_inline_fast_path_with_plan(range, object_info, fi, opts, ReadPathPlan::new(object_info, fi))
}
fn should_use_inline_fast_path_with_plan(
range: &Option<HTTPRangeSpec>,
object_info: &ObjectInfo,
fi: &FileInfo,
opts: &ObjectOptions,
plan: ReadPathPlan,
) -> bool {
if !object_info.is_inline_fast_path_eligible() || fi.data.is_none() || range.is_some() || opts.part_number.is_some() {
return false;
}
let Some(shape) = plan.shape() else {
return false;
};
// The persisted marker is authoritative for the storage decision, but it
// is still untrusted metadata at this boundary. Revalidate its geometry so
// a stale/corrupt marker cannot route an oversized payload into the
// in-memory decoder. The independent direct-memory reader remains capped
// at 128 KiB below.
if !plan.is_plain()
|| shape.object_size > usize::try_from(INLINE_FAST_PATH_MAX_OBJECT_SIZE).expect("inline fast path limit fits usize")
|| fi.erasure.data_blocks == 0
|| fi.erasure.block_size == 0
{
return false;
}
true
}
enum SmallWritePath {
@@ -4817,6 +5040,28 @@ fn should_force_delete_marker_for_missing_version(opts: &ObjectOptions) -> bool
opts.delete_marker || ((opts.versioned || opts.version_suspended) && opts.version_id.is_none() && !opts.data_movement)
}
/// Whether a plain client delete addressed to an explicit version removed an
/// existing delete marker, so the response must carry delete-marker semantics
/// (`x-amz-delete-marker: true` / `DeleteMarker` in `DeleteObjects` entries).
///
/// This is a response-side classification only. It must never feed the
/// storage write shape: a delete request marked `deleted` with the null
/// identity makes `FileMeta::delete_version` re-create the marker it just
/// removed (that is the suspended-bucket "delete mints a marker" write
/// path), which is why `resolve_delete_version_state` cannot simply report
/// `delete_marker` for suspended buckets (issue #6745).
///
/// Purge/replica replication shapes are excluded, mirroring the clauses in
/// `resolve_delete_version_state` that clear `delete_marker` for them.
fn explicit_delete_removed_marker(opts: &ObjectOptions, goi: &ObjectInfo, version_found: bool) -> bool {
opts.version_id.is_some()
&& version_found
&& goi.delete_marker
&& goi.version_purge_status.is_empty()
&& opts.version_purge_status().is_empty()
&& opts.delete_marker_replication_status().is_empty()
}
fn resolve_delete_version_state(opts: &ObjectOptions, goi: &ObjectInfo, version_found: bool) -> (bool, bool) {
let mut mark_delete = goi.version_id.is_some() || ((opts.versioned || opts.version_suspended) && opts.version_id.is_none());
let mut delete_marker = opts.versioned;
@@ -7038,6 +7283,85 @@ mod tests {
);
}
#[test]
fn resolve_delete_version_state_keeps_null_marker_removal_write_shape_undeleted() {
// Removing a null delete marker by explicit version id on a
// versioning-suspended bucket must NOT mark the write request
// `deleted`: `FileMeta::delete_version` re-creates a marker for
// `deleted` requests carrying the null identity (the suspended-bucket
// "delete mints a marker" path), which would resurrect the marker
// being removed. The response-side marker semantics come from
// `explicit_delete_removed_marker` instead (issue #6745).
let opts = ObjectOptions {
version_suspended: true,
version_id: Some(Uuid::nil().to_string()),
..Default::default()
};
let current = ObjectInfo {
version_id: Some(Uuid::nil()),
delete_marker: true,
..Default::default()
};
let (mark_delete, delete_marker) = resolve_delete_version_state(&opts, &current, true);
assert!(!mark_delete);
assert!(!delete_marker, "the storage write for a null-marker removal must stay undeleted");
assert!(
explicit_delete_removed_marker(&opts, &current, true),
"the response must still report delete-marker semantics"
);
}
#[test]
fn explicit_delete_removed_marker_is_limited_to_plain_marker_removals() {
let opts = ObjectOptions {
versioned: true,
version_id: Some(Uuid::new_v4().to_string()),
..Default::default()
};
let marker = ObjectInfo {
version_id: Some(Uuid::new_v4()),
delete_marker: true,
..Default::default()
};
assert!(explicit_delete_removed_marker(&opts, &marker, true));
assert!(
!explicit_delete_removed_marker(&opts, &marker, false),
"missing versions are not marker removals"
);
let data_version = ObjectInfo {
version_id: marker.version_id,
delete_marker: false,
..Default::default()
};
assert!(!explicit_delete_removed_marker(&opts, &data_version, true));
let versionless = ObjectOptions {
versioned: true,
..Default::default()
};
assert!(
!explicit_delete_removed_marker(&versionless, &marker, true),
"marker creation (no version in the request) is not a removal"
);
let replica_purge = ObjectOptions {
versioned: true,
version_id: Some(Uuid::new_v4().to_string()),
delete_replication: Some(ReplicationState {
replica_status: ReplicationStatusType::Replica,
..Default::default()
}),
..Default::default()
};
assert!(
!explicit_delete_removed_marker(&replica_purge, &marker, true),
"replication purge shapes keep their existing response semantics"
);
}
#[test]
fn resolve_delete_version_state_keeps_delete_marker_for_replica_marker_creation() {
let opts = ObjectOptions {
@@ -10881,6 +11205,41 @@ mod tests {
));
}
#[test]
fn read_path_shape_rejects_mismatched_metadata_copies_and_transforms() {
let (object_info, fi, _) = direct_memory_test_metadata(1024);
let shape = read_path_shape(&object_info, &fi).expect("matching metadata should have a valid shape");
assert_eq!(shape.object_size, 1024);
assert!(shape.is_plain(&object_info, &fi));
let mut bad_part_number = object_info.clone();
Arc::make_mut(&mut bad_part_number.parts)[0].number = 2;
assert!(read_path_shape(&bad_part_number, &fi).is_none());
let mut bad_part_size = fi.clone();
bad_part_size.parts[0].size += 1;
assert!(read_path_shape(&object_info, &bad_part_size).is_none());
let mut bad_actual_size = object_info.clone();
Arc::make_mut(&mut bad_actual_size.parts)[0].actual_size += 1;
assert!(read_path_shape(&bad_actual_size, &fi).is_none());
let mut compressed = fi.clone();
insert_str(&mut compressed.metadata, SUFFIX_COMPRESSION, "zstd".to_string());
let compressed_shape = read_path_shape(&object_info, &compressed).expect("geometry remains valid");
assert!(!compressed_shape.is_plain(&object_info, &compressed));
let mut encrypted = fi;
encrypted
.metadata
.insert("x-amz-server-side-encryption".to_string(), "AES256".to_string());
assert!(
!read_path_shape(&object_info, &encrypted)
.expect("geometry remains valid")
.is_plain(&object_info, &encrypted)
);
}
#[test]
fn inline_fast_path_rejects_part_number_requests() {
let (mut object_info, mut fi, opts) = direct_memory_test_metadata(1024);
@@ -10894,6 +11253,68 @@ mod tests {
assert!(!should_use_inline_fast_path(&None, &object_info, &fi, &part_opts));
}
#[test]
fn inline_fast_path_allows_layout_specific_ec8_and_ec12_256kib_objects() {
for (data_blocks, parity_blocks) in [(8, 4), (12, 4)] {
let object_size = 256 * 1024_i64;
let mut fi = FileInfo::new("bucket/object", data_blocks, parity_blocks);
fi.size = object_size;
fi.data = Some(Bytes::from_static(b"payload"));
fi.add_object_part(1, String::new(), object_size as usize, None, object_size, None, None);
let object_info = ObjectInfo {
size: object_size,
data_blocks,
parity_blocks,
inlined: true,
parts: Arc::new(fi.parts.clone()),
..Default::default()
};
assert!(should_use_inline_fast_path(&None, &object_info, &fi, &ObjectOptions::default()));
}
}
#[test]
fn inline_fast_path_rejects_oversized_or_corrupt_inline_markers() {
for (data_blocks, parity_blocks) in [(1, 0), (8, 4), (12, 4)] {
let object_size = 4 * 1024 * 1024_i64;
let mut fi = FileInfo::new("bucket/object", data_blocks, parity_blocks);
fi.size = object_size;
fi.data = Some(Bytes::from_static(b"payload"));
fi.add_object_part(1, String::new(), object_size as usize, None, object_size, None, None);
let object_info = ObjectInfo {
size: object_size,
data_blocks,
parity_blocks,
inlined: true,
parts: Arc::new(fi.parts.clone()),
..Default::default()
};
assert!(
object_info.is_inline_fast_path_eligible(),
"the marker and object shape alone must not be the final trust boundary"
);
assert!(!should_use_inline_fast_path(&None, &object_info, &fi, &ObjectOptions::default()));
}
// A malformed erasure geometry must fail closed before shard-size
// arithmetic, even when an inline marker and payload are present.
let (mut object_info, mut fi, opts) = direct_memory_test_metadata(4 * 1024 * 1024);
object_info.inlined = true;
fi.data = Some(Bytes::from_static(b"payload"));
fi.erasure.data_blocks = 0;
assert!(!should_use_inline_fast_path(&None, &object_info, &fi, &opts));
// A mismatched plain part size must not allow the large object size to
// reach the inline decoder's `Vec::with_capacity` allocation.
let (mut object_info, mut fi, opts) = direct_memory_test_metadata(4 * 1024 * 1024);
object_info.inlined = true;
fi.data = Some(Bytes::from_static(b"payload"));
fi.parts[0].actual_size = 0;
assert!(!should_use_inline_fast_path(&None, &object_info, &fi, &opts));
}
#[test]
fn small_object_direct_memory_decision_reports_bounded_reasons() {
let (object_info, fi, opts) = direct_memory_test_metadata(1024);
@@ -10967,6 +11388,13 @@ mod tests {
get_small_object_direct_memory_decision_with_threshold(&None, &object_info, &fi, &opts, true, 128 * 1024),
GetDirectMemoryDecision::Use { object_size: 1024 }
);
let mut corrupt_part = fi;
corrupt_part.parts[0].actual_size += 1;
assert_eq!(
get_small_object_direct_memory_decision_with_threshold(&None, &object_info, &corrupt_part, &opts, true, 128 * 1024),
GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::MetadataShape)
);
}
#[test]
+14 -1
View File
@@ -3543,7 +3543,20 @@ mod heal_result_report_tests {
.await
.expect("grace-protected dangling metadata should return a typed heal result");
assert_eq!(error, Some(DiskError::ErasureReadQuorum));
let error = error.expect("grace-protected dangling metadata should be reported as deferred");
assert!(
error.is_dangling_delete_grace(),
"grace-protected dangling metadata should keep a typed deferred-cleanup marker: {error}"
);
let message = error.to_string();
assert!(
message.contains("dangling object deletion deferred by heal grace window"),
"grace-protected dangling metadata should explain that cleanup was deferred: {message}"
);
assert!(
message.contains("retry_after_secs="),
"grace-protected dangling metadata should include retry timing: {message}"
);
assert!(
temp_dirs[0]
.path()
+285 -34
View File
@@ -84,11 +84,13 @@ use rustfs_rio::TryGetIndex;
use rustfs_utils::http::SSEC_ALGORITHM_HEADER;
#[cfg(test)]
use rustfs_utils::http::SUFFIX_COMPRESSION;
use rustfs_utils::http::{SUFFIX_MAX_TOTAL_OBJECT_SIZE, get_consistent_str};
use std::future::Future;
#[cfg(test)]
use std::sync::atomic::AtomicBool;
#[cfg(any(test, feature = "test-util"))]
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Mutex, OnceLock};
#[cfg(any(test, feature = "test-util"))]
use std::time::Duration;
#[cfg(test)]
@@ -97,6 +99,83 @@ use tokio::task::JoinSet;
const MULTIPART_LIST_IO_CONCURRENCY: usize = 16;
static CAPPED_MULTIPART_STAGING: OnceLock<Mutex<HashMap<String, Arc<tokio::sync::Semaphore>>>> = OnceLock::new();
struct CappedMultipartStagingGuard {
upload_id_path: String,
permit: Option<tokio::sync::OwnedSemaphorePermit>,
}
impl Drop for CappedMultipartStagingGuard {
fn drop(&mut self) {
// Release the permit before checking the Arc count so a concurrent
// Abort/Complete cleanup can remove the now-unused map entry.
self.permit.take();
remove_capped_multipart_staging_semaphore(&self.upload_id_path);
}
}
fn capped_multipart_staging_semaphore(upload_id_path: &str) -> Arc<tokio::sync::Semaphore> {
CAPPED_MULTIPART_STAGING
.get_or_init(|| Mutex::new(HashMap::new()))
.lock()
.expect("capped multipart staging semaphore map should not be poisoned")
.entry(upload_id_path.to_owned())
.or_insert_with(|| Arc::new(tokio::sync::Semaphore::new(1)))
.clone()
}
fn remove_capped_multipart_staging_semaphore(upload_id_path: &str) {
if let Some(map) = CAPPED_MULTIPART_STAGING.get() {
let mut map = map
.lock()
.expect("capped multipart staging semaphore map should not be poisoned");
let removable = map
.get(upload_id_path)
.is_some_and(|semaphore| Arc::strong_count(semaphore) == 1);
if removable {
map.remove(upload_id_path);
}
}
}
fn multipart_size_limit_from_metadata(metadata: &HashMap<String, String>) -> Result<Option<u64>> {
if !contains_key_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE) {
return Ok(None);
}
let Some(value) = get_consistent_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE) else {
return Err(Error::InvalidArgument(
"multipart upload".to_string(),
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
"missing or conflicting internal size limit".to_string(),
));
};
let limit = value.parse::<u64>().map_err(|_| {
Error::InvalidArgument(
"multipart upload".to_string(),
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
"invalid internal size limit".to_string(),
)
})?;
Ok(Some(limit))
}
fn admitted_multipart_size(current: u64, candidate: u64, limit: u64) -> Result<u64> {
let total = current.checked_add(candidate).ok_or_else(|| {
Error::InvalidArgument(
"multipart upload".to_string(),
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
"logical size overflow".to_string(),
)
})?;
if total > limit {
return Err(Error::EntityTooLarge(total, limit));
}
Ok(total)
}
pub(crate) struct StaleMultipartCleanupGuard {
file_info: FileInfo,
upload_path: String,
@@ -115,8 +194,13 @@ impl StaleMultipartCleanupGuard {
pub(crate) async fn delete(self, set: &SetDisks) -> Result<()> {
fence_commit_on_lock_loss(Some(&self.lock_guard), "stale_multipart_cleanup", &self.upload_path)?;
set.delete_all_with_quorum(RUSTFS_META_MULTIPART_BUCKET, &self.upload_path, self.write_quorum)
.await
let result = set
.delete_all_with_quorum(RUSTFS_META_MULTIPART_BUCKET, &self.upload_path, self.write_quorum)
.await;
if result.is_ok() {
remove_capped_multipart_staging_semaphore(&self.upload_path);
}
result
}
}
@@ -635,6 +719,61 @@ async fn multipart_upload_paths_on_disk(disk: DiskStore, bucket: &str) -> disk::
}
impl SetDisks {
async fn current_multipart_logical_size(
&self,
bucket: &str,
object: &str,
upload_id: &str,
upload_id_path: &str,
fi: &FileInfo,
replacing_part: usize,
) -> Result<u64> {
let online_disks = self.get_disks_internal().await;
let read_quorum = fi.read_quorum(self.default_read_quorum());
let part_path = format!(
"{}{}",
path_join_buf(&[
upload_id_path,
fi.data_dir.map(|v| v.to_string()).unwrap_or_default().as_str(),
]),
SLASH_SEPARATOR
);
let part_numbers = match Self::list_parts(&online_disks, &part_path, read_quorum).await {
Ok(parts) => parts,
Err(DiskError::FileNotFound) => return Ok(0),
Err(err) => return Err(to_object_err(err.into(), vec![bucket, object, upload_id])),
};
if part_numbers.is_empty() {
return Ok(0);
}
let part_meta_paths = part_numbers
.iter()
.map(|number| format!("{part_path}part.{number}.meta"))
.collect::<Vec<_>>();
let existing_parts =
Self::read_parts(&online_disks, RUSTFS_META_MULTIPART_BUCKET, &part_meta_paths, &part_numbers, read_quorum)
.await
.map_err(|err| to_object_err(err.into(), vec![bucket, object, upload_id]))?;
existing_parts.into_iter().try_fold(0_u64, |total, part| {
if part.error.is_some() || part.number == replacing_part {
return if part.error.is_some() {
Err(Error::PartMissingOrCorrupt)
} else {
Ok(total)
};
}
let part_size = u64::try_from(part.actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
total.checked_add(part_size).ok_or_else(|| {
Error::InvalidArgument(
"multipart upload".to_string(),
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
"logical size overflow".to_string(),
)
})
})
}
async fn discover_multipart_upload_paths(
&self,
orig_bucket: &str,
@@ -1130,9 +1269,18 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
crate::hp_guard!("SetDisks::put_object_part");
let upload_id_path = Self::get_multipart_upload_dir(bucket, object, upload_id, opts.data_movement);
let (fi, _) = self
let (fi, _) = match self
.check_upload_id_exists_with_opts(bucket, object, upload_id, true, opts)
.await?;
.await
{
Ok(value) => value,
Err(err @ Error::InvalidUploadID(..)) => {
remove_capped_multipart_staging_semaphore(&upload_id_path);
return Err(err);
}
Err(err) => return Err(err),
};
let multipart_size_limit = multipart_size_limit_from_metadata(&fi.metadata)?;
ensure_data_movement_upload_access(&fi, bucket, object, upload_id, opts)?;
ensure_multipart_bucket_incarnation(&self.ctx, &fi, bucket, object, upload_id, opts.expected_bucket_incarnation_id)
.await?;
@@ -1165,6 +1313,44 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
let part_suffix = format!("part.{part_id}");
let tmp_part = format!("{}x{}", Uuid::new_v4(), OffsetDateTime::now_utc().unix_timestamp());
let tmp_part_path = Arc::new(format!("{tmp_part}/{part_suffix}"));
let part_path = format!("{}/{}/{}", upload_id_path, fi.data_dir.unwrap_or_default(), part_suffix);
let part_lock_path = format!("{upload_id_path}/{part_suffix}");
// Keep at most one capped part staging locally per upload. The
// distributed lock below is held only for the durable admission check;
// it is reacquired for the short final rename, so Complete/Abort are
// not blocked behind a slow body upload.
let _capped_staging_guard = if multipart_size_limit.is_some() {
Some(CappedMultipartStagingGuard {
upload_id_path: upload_id_path.clone(),
permit: Some(
capped_multipart_staging_semaphore(&upload_id_path)
.acquire_owned()
.await
.map_err(|_| Error::other("capped multipart staging semaphore closed"))?,
),
})
} else {
None
};
if let Some(limit) = multipart_size_limit {
let admission_guard = self
.acquire_write_lock_diag("put_object_part_admission", RUSTFS_META_MULTIPART_BUCKET, &upload_id_path)
.await?;
let declared_size = if data.size() >= 0 {
u64::try_from(data.size()).map_err(|_| Error::PartMissingOrCorrupt)?
} else if data.actual_size() >= 0 {
u64::try_from(data.actual_size()).map_err(|_| Error::PartMissingOrCorrupt)?
} else {
return Err(Error::PartMissingOrCorrupt);
};
let current_size = self
.current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &fi, part_id)
.await?;
admitted_multipart_size(current_size, declared_size, limit)?;
drop(admission_guard);
}
let result: Result<PartInfo> = async {
let erasure =
@@ -1365,30 +1551,21 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
.await?;
}
let part_path = format!("{}/{}/{}", upload_id_path, fi.data_dir.unwrap_or_default(), part_suffix);
let part_lock_path = format!("{upload_id_path}/{part_suffix}");
#[cfg(test)]
pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockAcquire).await;
// Serialize only same-part commits (rename_part), not the whole upload.
// Each concurrent stream writes to its own unique temp dir (see
// `tmp_part` above), so the encode/stream phase never conflicts and must
// stay lock-free — holding a lock across it would serialize slow
// re-transmits of the same part and defeat the S3 "last finisher wins"
// semantics. The mixed-generation hazard is confined to rename_part,
// where two temp parts are moved cross-disk onto the SAME final
// part_path: interleaving there can leave shards from two generations,
// each individually bitrot-valid, that only surface as silent corruption
// at read time (backlog#853). A write lock scoped to this part number
// makes each same-part commit atomic across disks, so the last committer
// wins consistently, while different part numbers commit onto disjoint
// part paths and stay concurrent (issue#5961 — an uploadId-wide write
// lock serialized them into 503 lock-acquire timeouts). The shared
// uploadId read lock keeps completion/abort (which take the uploadId
// write lock) from racing any in-flight part commit; a guarded
// completion takes the object lock before the upload lock to preserve
// global ordering.
let (_upload_commit_guard, _part_commit_guard) = if opts.no_lock {
// Capped uploads reacquire the upload-wide write lock for the
// final durable check and rename. Uncapped uploads retain the
// concurrent encode path and only serialize the final same-part
// rename; completion/abort use the upload-wide write lock.
let (_upload_commit_guard, _part_commit_guard) = if multipart_size_limit.is_some() {
let upload_guard = self
.acquire_write_lock_diag("put_object_part_commit", RUSTFS_META_MULTIPART_BUCKET, &upload_id_path)
.await?;
let part_guard = self
.acquire_write_lock_diag("put_object_part_commit", RUSTFS_META_MULTIPART_BUCKET, &part_lock_path)
.await?;
(Some(upload_guard), Some(part_guard))
} else if opts.no_lock {
(None, None)
} else {
let upload_guard = self
@@ -1400,8 +1577,16 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
(Some(upload_guard), Some(part_guard))
};
let (commit_fi, _) = self
.check_upload_id_exists_with_opts(bucket, object, upload_id, false, opts)
.check_upload_id_exists_with_opts(bucket, object, upload_id, multipart_size_limit.is_some(), opts)
.await?;
let commit_size_limit = multipart_size_limit_from_metadata(&commit_fi.metadata)?;
if commit_size_limit != multipart_size_limit {
return Err(Error::InvalidArgument(
"multipart upload".to_string(),
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
"size limit metadata changed or is missing".to_string(),
));
}
ensure_data_movement_upload_access(&commit_fi, bucket, object, upload_id, opts)?;
ensure_multipart_bucket_incarnation(
&self.ctx,
@@ -1431,6 +1616,14 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
}
ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
if let Some(limit) = commit_size_limit {
let current_size = self
.current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &commit_fi, part_id)
.await?;
let candidate_size = u64::try_from(actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
admitted_multipart_size(current_size, candidate_size, limit)?;
}
let _ = self
.rename_part(
&shuffle_disks,
@@ -1891,12 +2084,17 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
let upload_id_path = Self::get_multipart_upload_dir(bucket, object, upload_id, opts.data_movement);
self.delete_all_with_quorum(
RUSTFS_META_MULTIPART_BUCKET,
&upload_id_path,
fi.write_quorum(self.default_write_quorum()),
)
.await
let result = self
.delete_all_with_quorum(
RUSTFS_META_MULTIPART_BUCKET,
&upload_id_path,
fi.write_quorum(self.default_write_quorum()),
)
.await;
if result.is_ok() {
remove_capped_multipart_staging_semaphore(&upload_id_path);
}
result
}
// complete_multipart_upload finished
#[tracing::instrument(skip(self))]
@@ -2016,6 +2214,27 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
return Err(Error::other("part result number err"));
}
if let Some(limit) = multipart_size_limit_from_metadata(&fi.metadata)? {
let mut total = 0_u64;
for part in &object_parts {
if part.error.is_some() {
return Err(Error::PartMissingOrCorrupt);
}
let part_size = u64::try_from(part.actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
total = total.checked_add(part_size).ok_or_else(|| {
Error::InvalidArgument(
"multipart upload".to_string(),
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
"logical size overflow".to_string(),
)
})?;
}
if total > limit {
return Err(Error::EntityTooLarge(total, limit));
}
rustfs_utils::http::metadata_compat::remove_str(&mut fi.metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE);
}
let mut checksum_type = rustfs_rio::ChecksumType::NONE;
if let Some(cs) = fi.metadata.get(rustfs_rio::RUSTFS_MULTIPART_CHECKSUM) {
@@ -3024,13 +3243,17 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
Ok(ObjectInfo::from_file_info(&fi, &commit_bucket, &commit_object, commit_is_versioned))
};
if detach_commit_owner {
let result = if detach_commit_owner {
tokio::spawn(commit)
.await
.map_err(|err| Error::other(format!("complete_multipart_upload commit task failed: {err}")))?
} else {
commit.await
};
if result.is_ok() {
remove_capped_multipart_staging_semaphore(&upload_id_path);
}
result
}
}
@@ -3096,6 +3319,34 @@ mod tests {
assert!(multipart_bucket_incarnation_id(&nil_metadata).is_err());
}
#[test]
fn multipart_size_limit_metadata_is_dual_key_and_fail_closed() {
let mut metadata = HashMap::new();
insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "100".to_string());
assert_eq!(multipart_size_limit_from_metadata(&metadata).unwrap(), Some(100));
metadata.insert("x-minio-internal-max-total-object-size".to_string(), "101".to_string());
assert!(multipart_size_limit_from_metadata(&metadata).is_err());
let mut invalid = HashMap::new();
insert_str(&mut invalid, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "-1".to_string());
assert!(multipart_size_limit_from_metadata(&invalid).is_err());
}
#[test]
fn multipart_size_admission_handles_boundaries_and_overflow() {
assert_eq!(admitted_multipart_size(90, 10, 100).unwrap(), 100);
assert!(matches!(
admitted_multipart_size(90, 11, 100),
Err(StorageError::EntityTooLarge(101, 100))
));
assert!(admitted_multipart_size(u64::MAX - 1, 1, u64::MAX).is_ok());
assert!(matches!(
admitted_multipart_size(u64::MAX, 1, u64::MAX),
Err(StorageError::InvalidArgument(_, _, _))
));
}
#[test]
fn multipart_bucket_incarnation_gate_accepts_only_current_or_same_lifetime_legacy_uploads() {
let expected = Uuid::new_v4();
+661 -69
View File
@@ -30,31 +30,34 @@ use super::super::{
GET_OBJECT_PATH_DIRECT_MEMORY, GET_OBJECT_PATH_EMPTY, GET_OBJECT_PATH_INLINE_DIRECT, GET_OBJECT_PATH_INTERNAL_META,
GET_OBJECT_PATH_LEGACY_DUPLEX, GET_OBJECT_PATH_REMOTE_TRANSITION, GET_OBJECT_PATH_SET_DISK, GET_STAGE_DECODE, GET_STAGE_EMIT,
GET_STAGE_INLINE_PREPARE, GET_STAGE_LOCK_ACQUIRE, GET_STAGE_METADATA, GET_STAGE_OBJECT_INFO, GET_STAGE_PATH_DECISION,
GET_STAGE_READER_SETUP, GenericError, GetCodecStreamingDecision, GetDirectMemoryDecision, GetObjectReader, HTTPRangeSpec,
HashAlgorithm, HashMap, HashReader, HashSet, HeaderMap, HealChannelPriority, InstanceContext, Instant, LOG_COMPONENT_ECSTORE,
LOG_SUBSYSTEM_SET_DISK, OBJECT_OP_IGNORED_ERRS, ObjectApiError, ObjectInfo, ObjectKey, ObjectLockConfigSnapshot,
ObjectLockConfigState, ObjectOptions, ObjectReader, ObjectToDelete, OffsetDateTime, Ordering, Pin, PutObjReader,
RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET, ReaderImpl, ReplicateDecision, ReplicationObjectBridge, Result,
SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS, SLASH_SEPARATOR, SUFFIX_ACTUAL_SIZE, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE,
SUFFIX_RESTORE_OPERATION_ID, SetDisks, SmallWritePath, StorageError, TRANSITION_COMPLETE, UpdateMetadataOpts, Uuid,
WriteLayout, X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, X_AMZ_RESTORE,
adaptive_duplex_buffer_size, build_get_object_info, build_inline_bitrot_readers, build_inline_bitrot_readers_from_refs,
can_try_inline_data_shards_direct, check_object_lock_delete, check_object_lock_for_deletion_with_state,
check_object_lock_retention_update, classify_get_codec_streaming_object_class, classify_put_write_path,
classify_storage_error, collect_inline_data_shard_fileinfos_by_index, contains_key_str, create_bitrot_writer, debug,
delete_file_info_version_id, disk, ensure_delete_commit_locks_held, error, finish_set_disk_read_lock,
get_codec_streaming_reader_gate, get_object_body_cache_hook, get_raw_etag, get_small_object_direct_memory_decision,
GET_STAGE_READER_SETUP, GenericError, GetCodecStreamingDecision, GetCodecStreamingFallbackReason, GetDirectMemoryDecision,
GetObjectReader, HTTPRangeSpec, HashAlgorithm, HashMap, HashReader, HashSet, HeaderMap, HealChannelPriority, InstanceContext,
Instant, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_SET_DISK, OBJECT_OP_IGNORED_ERRS, ObjectApiError, ObjectInfo, ObjectKey,
ObjectLockConfigSnapshot, ObjectLockConfigState, ObjectOptions, ObjectReader, ObjectToDelete, OffsetDateTime, Ordering, Pin,
PutObjReader, RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET, ReadPathPlan, ReaderImpl, ReplicateDecision,
ReplicationObjectBridge, Result, SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS, SLASH_SEPARATOR, SUFFIX_ACTUAL_SIZE,
SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_RESTORE_OPERATION_ID, SetDisks, SmallWritePath, StorageError,
TRANSITION_COMPLETE, UpdateMetadataOpts, Uuid, WriteLayout, X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE,
X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, X_AMZ_RESTORE, adaptive_duplex_buffer_size, build_get_object_info,
build_inline_bitrot_readers, build_inline_bitrot_readers_from_refs, can_try_inline_data_shards_direct,
check_object_lock_delete, check_object_lock_for_deletion_with_state, check_object_lock_retention_update,
classify_get_codec_streaming_object_class, classify_put_write_path, classify_storage_error,
collect_inline_data_shard_fileinfos_by_index, contains_key_str, create_bitrot_writer, debug, delete_file_info_version_id,
disk, ensure_delete_commit_locks_held, error, explicit_delete_removed_marker, finish_set_disk_read_lock,
get_codec_streaming_reader_gate_with_plan, get_object_body_cache_hook, get_raw_etag,
get_small_object_direct_memory_decision_with_threshold_and_plan, get_small_object_direct_memory_threshold,
get_stage_timer_if_enabled, get_str, get_transitioned_object_reader_with_tier_manager, inline_erasure_shard_file_offset,
inline_erasure_shard_size, insert_str, is_deadlock_detection_enabled, is_err_object_not_found, is_err_version_not_found,
is_explicit_null_version, is_lock_optimization_enabled, issue3031_diag_enabled, join_all, known_put_object_storage_size,
path_join_buf, put_restore_opts, record_compression_total_memory, record_get_codec_streaming_gate_decision,
is_explicit_null_version, is_get_codec_streaming_base_enabled, is_get_small_object_direct_memory_enabled,
is_lock_optimization_enabled, issue3031_diag_enabled, join_all, known_put_object_storage_size, path_join_buf,
put_restore_opts, record_compression_total_memory, record_get_codec_streaming_gate_decision,
record_get_direct_memory_decision, record_get_object_pipeline_failure, record_get_object_pipeline_failure_for_path,
record_get_object_reader_path_observation, record_get_stage_duration_if_enabled, record_lock_acquire,
reduce_write_quorum_errs, release_materialized_read_lock, replication_write_may_pass_worm_gate, require_restore_operation_id,
resolve_delete_version_state, resolve_tiered_decommission_write_quorum_result, resolve_write_layout,
restore_commit_operation_id_from_metadata, restore_operation_id_from_metadata, send_event,
set_disk_delete_creates_delete_marker, should_force_delete_marker_for_missing_version,
should_persist_encryption_original_size, should_preserve_delete_replication_state, should_use_inline_fast_path,
should_persist_encryption_original_size, should_preserve_delete_replication_state, should_use_inline_fast_path_with_plan,
take_prepared_get_object_metadata, to_object_err, try_read_inline_data_shards_direct, warn,
};
use super::bitrot_self_verify::{BitrotSelfVerifyTarget, drop_failed_writer_disks, verify_written_bitrot_shards};
@@ -67,7 +70,7 @@ use crate::set_disk::coding;
use crate::set_disk::core::io_primitives::GetCodecStreamingReaderBuildOutcome;
use crate::set_disk::mem;
use crate::set_disk::metadata_sys;
use crate::set_disk::read::GetObjectDownstreamWriter;
use crate::set_disk::read::{GET_OBJECT_PATH_MID_SIZE_STREAMING, GetObjectDownstreamWriter};
use crate::set_disk::runtime_sources;
use crate::storage_api_contracts::multipart::MultipartOperations;
use crate::storage_api_contracts::object::ObjectIO;
@@ -79,6 +82,111 @@ use rustfs_rio::TryGetIndex;
use rustfs_utils::http::HeaderExt;
use tokio::io::AsyncWriteExt;
// Keep the mid-size reader separate from the codec-streaming rollout. The
// latter is intentionally opt-in because its per-stripe worker can regress
// tiny objects; this bounded range is the gap between direct-memory GETs and
// the legacy duplex reader.
const ENV_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE: &str = "RUSTFS_GET_MID_SIZE_STREAMING_ENABLE";
const DEFAULT_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE: bool = true;
const GET_MID_SIZE_STREAMING_MIN_SIZE: usize = 128 * 1024 + 1;
const GET_MID_SIZE_STREAMING_MAX_SIZE: usize = 1024 * 1024;
fn is_get_mid_size_streaming_enabled() -> bool {
#[cfg(test)]
{
rustfs_utils::get_env_bool(ENV_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE, DEFAULT_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE)
}
#[cfg(not(test))]
{
static ENABLED: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
*ENABLED.get_or_init(|| {
rustfs_utils::get_env_bool(ENV_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE, DEFAULT_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE)
})
}
}
/// Return the object size when the bounded non-duplex reader is safe.
///
/// This predicate deliberately has a narrower contract than the general
/// codec-streaming gate: only a whole, plain, single-part object is eligible.
/// Ranges, transforms, remote objects, multipart reads, copy-source reads and
/// special movement/version requests retain their existing legacy semantics.
#[allow(dead_code, reason = "asserted by this file's gate tests")]
fn get_mid_size_streaming_object_size(
range: &Option<HTTPRangeSpec>,
object_info: &ObjectInfo,
fi: &FileInfo,
opts: &ObjectOptions,
lock_optimization_enabled: bool,
) -> Option<usize> {
get_mid_size_streaming_object_size_with_flags(
range,
object_info,
fi,
opts,
lock_optimization_enabled,
is_get_mid_size_streaming_enabled(),
is_get_codec_streaming_base_enabled(),
)
}
#[allow(dead_code, reason = "asserted by this file's gate tests")]
fn get_mid_size_streaming_object_size_with_flags(
range: &Option<HTTPRangeSpec>,
object_info: &ObjectInfo,
fi: &FileInfo,
opts: &ObjectOptions,
lock_optimization_enabled: bool,
mid_size_enabled: bool,
codec_base_enabled: bool,
) -> Option<usize> {
get_mid_size_streaming_object_size_with_flags_and_plan(
range,
object_info,
opts,
lock_optimization_enabled,
mid_size_enabled,
codec_base_enabled,
super::super::ReadPathPlan::new(object_info, fi),
)
}
fn get_mid_size_streaming_object_size_with_flags_and_plan(
range: &Option<HTTPRangeSpec>,
object_info: &ObjectInfo,
opts: &ObjectOptions,
lock_optimization_enabled: bool,
mid_size_enabled: bool,
codec_base_enabled: bool,
plan: super::super::ReadPathPlan,
) -> Option<usize> {
if !mid_size_enabled
|| !codec_base_enabled
|| !lock_optimization_enabled
|| range.is_some()
|| opts.part_number.is_some()
|| opts.version_id.is_some()
|| opts.incl_free_versions
|| opts.skip_free_version
|| opts.data_movement
|| opts.raw_data_movement_read
|| object_info.delete_marker
|| object_info.metadata_only
|| object_info.version_only
|| crate::set_disk::get_object_read_policy() != super::super::GetObjectReadPolicy::Default
{
return None;
}
let shape = plan.shape()?;
if !plan.is_plain() {
return None;
}
(GET_MID_SIZE_STREAMING_MIN_SIZE..=GET_MID_SIZE_STREAMING_MAX_SIZE)
.contains(&shape.object_size)
.then_some(shape.object_size)
}
#[cfg(all(test, feature = "test-util"))]
use super::super::GetObjectMetadataCacheEntry;
#[cfg(test)]
@@ -1629,11 +1737,21 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
return Ok(reader);
}
// All remaining local fast paths share this immutable, fail-closed
// metadata decision. Build it once after empty/remote exits so those
// requests do not pay for part and transform scans they cannot use.
let read_path_plan = ReadPathPlan::new(&object_info, fi);
// Inline data fast path: skip duplex pipe for small inline objects.
// Uses the shared predicate from ObjectInfo; additionally checks that
// inline data is actually present and neither range nor partNumber is
// in flight.
if should_use_inline_fast_path(&range, &object_info, fi, opts) {
let use_inline_fast_path = object_info.is_inline_fast_path_eligible()
&& fi.data.is_some()
&& range.is_none()
&& opts.part_number.is_none()
&& should_use_inline_fast_path_with_plan(&range, &object_info, fi, opts, read_path_plan);
if use_inline_fast_path {
let mut inline_prepare_stage_start = get_stage_timer_if_enabled(stage_metrics_enabled);
let data_shards = fi.erasure.data_blocks;
@@ -1795,27 +1913,10 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
}
}
let path_decision_stage_start = get_stage_timer_if_enabled(stage_metrics_enabled);
let codec_streaming_gate = get_codec_streaming_reader_gate(
bucket,
object,
opts.part_number,
object_class,
&object_info,
fi,
lock_optimization_enabled,
);
record_get_stage_duration_if_enabled(GET_OBJECT_PATH_SET_DISK, GET_STAGE_PATH_DECISION, path_decision_stage_start);
if object_info.is_remote() {
if let GetCodecStreamingDecision::Fallback(reason) = codec_streaming_gate.decision {
record_get_codec_streaming_gate_decision(
codec_streaming_gate.object_class,
codec_streaming_gate.decision,
size_bucket,
);
rustfs_io_metrics::record_get_object_codec_streaming_fallback(reason.as_str());
}
let decision = GetCodecStreamingDecision::Fallback(GetCodecStreamingFallbackReason::Remote);
record_get_codec_streaming_gate_decision(object_class, decision, size_bucket);
rustfs_io_metrics::record_get_object_codec_streaming_fallback(GetCodecStreamingFallbackReason::Remote.as_str());
record_get_object_reader_path_observation(GET_OBJECT_PATH_REMOTE_TRANSITION, object_class, size_bucket);
let mut opts = opts.clone();
if object_info.parts.len() == 1 {
@@ -1835,6 +1936,10 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
return Ok(finish_set_disk_read_lock(gr, read_lock_guard.take(), bucket, object));
}
// Metadata resolution and the remote-tier branch are complete here.
// Keep the rollout/configuration gate deferred until the request
// really needs codec streaming so an opted-out codec path cannot add
// fixed cost to the inline/direct-memory/mid-size hot paths.
// App-layer object data cache probe: metadata (etag/size) is resolved
// but no data shards have been read yet, so a hit skips the erasure
// read, bitrot verify and decode entirely. The hook validates object
@@ -1881,7 +1986,15 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
}
}
let direct_memory_decision = get_small_object_direct_memory_decision(&range, &object_info, fi, opts);
let direct_memory_decision = get_small_object_direct_memory_decision_with_threshold_and_plan(
&range,
&object_info,
fi,
opts,
is_get_small_object_direct_memory_enabled(),
get_small_object_direct_memory_threshold(),
read_path_plan,
);
record_get_direct_memory_decision(object_class, direct_memory_decision, size_bucket);
if let GetDirectMemoryDecision::Use { object_size } = direct_memory_decision {
if let Some(body) = Self::try_get_object_direct_data_shards_with_fileinfo(
@@ -1962,6 +2075,66 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
return Ok(reader);
}
// Mid-size plain objects use a bounded, non-duplex reader. Keep this
// path independent from the general codec-streaming rollout: that
// rollout remains off by default because its worker overhead is not a
// win for tiny objects. A failed setup degrades to the existing codec
// gate/legacy path before any response bytes are returned.
if get_mid_size_streaming_object_size_with_flags_and_plan(
&range,
&object_info,
opts,
lock_optimization_enabled,
is_get_mid_size_streaming_enabled(),
is_get_codec_streaming_base_enabled(),
read_path_plan,
)
.is_some()
{
match Self::get_object_mid_size_reader_with_fileinfo(
bucket,
object,
Arc::clone(&self.erasure_cache),
fi,
files,
disks,
self.set_index,
self.pool_index,
opts.skip_verify_bitrot,
object_class.as_str(),
size_bucket,
false,
)
.await?
{
GetCodecStreamingReaderBuildOutcome::Reader(stream) => {
record_get_object_reader_path_observation(GET_OBJECT_PATH_MID_SIZE_STREAMING, object_class, size_bucket);
let (mut reader, _offset, _length) =
get_object_reader_with_context(&self.ctx, stream, range, &object_info, opts, &h).await?;
reader.body_source = body_source;
return Ok(finish_set_disk_read_lock(reader, read_lock_guard.take(), bucket, object));
}
GetCodecStreamingReaderBuildOutcome::Fallback(_) => {
// The setup found a degraded-but-readable layout. Let the
// established codec gate and then legacy path handle it;
// this preserves whole-request fallback semantics.
}
}
}
let path_decision_stage_start = get_stage_timer_if_enabled(stage_metrics_enabled);
let codec_streaming_gate = get_codec_streaming_reader_gate_with_plan(
bucket,
object,
opts.part_number,
object_class,
&object_info,
fi,
lock_optimization_enabled,
read_path_plan,
);
record_get_stage_duration_if_enabled(GET_OBJECT_PATH_SET_DISK, GET_STAGE_PATH_DECISION, path_decision_stage_start);
match codec_streaming_gate.decision {
GetCodecStreamingDecision::Use => {
match Self::get_object_decode_reader_with_fileinfo(
@@ -2458,29 +2631,19 @@ impl SetDisks {
let mut object_lock_guard = None;
let mut bucket_lifecycle_guard = None;
let deferred_data_movement_precondition = opts.data_movement && opts.http_preconditions.is_some();
if opts.http_preconditions.is_some() && !deferred_data_movement_precondition {
if !opts.no_lock {
if let Some(expected_incarnation_id) = opts.expected_bucket_incarnation_id
&& opts.bucket_lifecycle_lock_fence.is_none()
{
bucket_lifecycle_guard = Some(
metadata_sys::object_store_in(&self.ctx)
.await?
.acquire_bucket_incarnation_fence(bucket, expected_incarnation_id)
.await?,
);
}
object_lock_guard = Some(
self.acquire_write_lock_diag("put_object_precondition", bucket, object)
.await?,
);
}
if let Some(err) = self.check_write_precondition(bucket, object, opts).await {
return Err(err);
}
// This pre-body check is advisory fast-fail only: the authoritative
// precondition evaluation happens under the commit namespace lock
// below, so the namespace write lock must NOT be taken here — holding
// it across client-paced body ingestion starves concurrent reads of
// the same object into lock-timeout 503s (rustfs/backlog#2074).
// Data movement skips the advisory read: its staleness predicate is
// only meaningful at commit time.
if opts.http_preconditions.is_some()
&& !opts.data_movement
&& let Some(err) = self.check_write_precondition(bucket, object, opts).await
{
return Err(err);
}
let expected_restore_operation_id = restore_commit_operation_id_from_metadata(&opts.user_defined)?;
@@ -2950,7 +3113,9 @@ impl SetDisks {
#[cfg(any(test, feature = "test-util"))]
pause_put_object_commit(bucket, object, PutObjectCommitPause::AfterNamespace).await;
if deferred_data_movement_precondition && let Some(err) = self.check_write_precondition(bucket, object, opts).await {
if opts.http_preconditions.is_some()
&& let Some(err) = self.check_write_precondition(bucket, object, opts).await
{
return Err(err);
}
@@ -6521,7 +6686,21 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
}
}
if goi.delete_marker && dobj.version_id.is_some() && goi.version_id == version_id {
// Same normalization as `explicit_delete_marker` above: `goi.version_id`
// is the client-facing identity (`Some(Uuid::nil())` for a null
// version) while `version_id` is the storage identity (`None` for an
// explicit null). Comparing them raw made a null delete marker's
// removal take the non-marker branch below, so the response lost
// `DeleteMarker`/`DeleteMarkerVersionId` and the removal was
// accounted as an object deletion (issue #6745).
let removed_delete_marker =
goi.delete_marker && dobj.version_id.is_some() && delete_file_info_version_id(goi.version_id) == version_id;
// Response-side only for the null identity: a delete request marked
// `deleted` with `version_id == None` makes `FileMeta::delete_version`
// re-create the marker it just removed (the suspended-bucket
// "delete mints a marker" write path), so the write shape must stay
// untouched for explicit null-marker removals.
if removed_delete_marker && version_id.is_some() {
vr.deleted = true;
vr.mod_time = goi.mod_time;
}
@@ -6540,11 +6719,21 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
}
};
if vr.deleted {
if vr.deleted || removed_delete_marker {
del_objects[i] = DeletedObject {
delete_marker: vr.deleted,
delete_marker_version_id: vr.version_id,
delete_marker_mtime: vr.mod_time,
delete_marker: true,
// `vr.version_id` holds the storage identity, which is
// `None` for an explicit null-marker removal; report the
// client-facing null identity so the response can carry
// `DeleteMarkerVersionId` for the marker that was removed.
delete_marker_version_id: if explicit_null_version {
Some(Uuid::nil())
} else {
vr.version_id
},
// For a null-marker removal `vr` stays undeleted (write
// shape), so take the marker's mtime from the source.
delete_marker_mtime: vr.mod_time.or(goi.mod_time),
object_name: vr.name.clone(),
replication_state: vr.replication_state_internal.clone(),
..Default::default()
@@ -7197,6 +7386,14 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
obj_info.user_defined = Arc::clone(&goi.user_defined);
obj_info.parts = Arc::clone(&goi.parts);
obj_info.user_tags = Arc::clone(&goi.user_tags);
// Report delete-marker semantics for an explicit-version delete whose
// target was a delete marker. On versioning-suspended buckets
// `resolve_delete_version_state` cannot mark the write request itself
// (a `deleted` write with the null identity re-creates the marker), so
// the marker-ness is restored on the response here (issue #6745).
if explicit_delete_removed_marker(&opts, &goi, version_found) {
obj_info.delete_marker = true;
}
self.invalidate_get_object_metadata_cache(bucket, object).await;
Ok(obj_info)
}
@@ -8165,6 +8362,190 @@ mod erasure_construction_tests {
}
}
#[cfg(test)]
mod mid_size_streaming_gate_tests {
use super::*;
use rustfs_filemeta::ObjectPartInfo;
use serial_test::serial;
fn plain_metadata(size: usize) -> (ObjectInfo, FileInfo) {
let size_i64 = i64::try_from(size).expect("test size fits i64");
let mut fi = FileInfo::new("object", 2, 2);
fi.size = size_i64;
fi.add_object_part(1, "etag".to_string(), size, fi.mod_time, size_i64, None, None);
let object_info = ObjectInfo {
size: size_i64,
parts: Arc::new(vec![ObjectPartInfo {
number: 1,
size,
actual_size: size_i64,
..Default::default()
}]),
..Default::default()
};
(object_info, fi)
}
#[test]
#[serial]
fn direct_memory_boundary_stays_out_of_mid_size_streaming() {
let (object_info, fi) = plain_metadata(128 * 1024);
assert_eq!(
get_mid_size_streaming_object_size_with_flags(&None, &object_info, &fi, &ObjectOptions::default(), true, true, true),
None
);
}
#[test]
#[serial]
fn mid_size_streaming_accepts_object_just_above_direct_memory_ceiling() {
let (object_info, fi) = plain_metadata(128 * 1024 + 1);
assert_eq!(
get_mid_size_streaming_object_size_with_flags(&None, &object_info, &fi, &ObjectOptions::default(), true, true, true),
Some(128 * 1024 + 1)
);
}
#[test]
#[serial]
fn mid_size_streaming_includes_one_mib_and_rejects_larger_objects() {
let (object_info, fi) = plain_metadata(1024 * 1024);
assert_eq!(
get_mid_size_streaming_object_size_with_flags(&None, &object_info, &fi, &ObjectOptions::default(), true, true, true),
Some(1024 * 1024)
);
let (large_info, large_fi) = plain_metadata(1024 * 1024 + 1);
assert_eq!(
get_mid_size_streaming_object_size_with_flags(
&None,
&large_info,
&large_fi,
&ObjectOptions::default(),
true,
true,
true
),
None
);
}
#[test]
#[serial]
fn mid_size_streaming_rejects_ranges_and_transformed_objects() {
let (object_info, fi) = plain_metadata(256 * 1024);
let range = Some(HTTPRangeSpec {
is_suffix_length: false,
start: 0,
end: 1,
});
assert_eq!(
get_mid_size_streaming_object_size_with_flags(&range, &object_info, &fi, &ObjectOptions::default(), true, true, true),
None
);
let mut encrypted = object_info;
encrypted.user_defined = Arc::new(HashMap::from([("x-minio-encryption-key".to_string(), "opaque".to_string())]));
assert_eq!(
get_mid_size_streaming_object_size_with_flags(&None, &encrypted, &fi, &ObjectOptions::default(), true, true, true),
None
);
}
#[test]
#[serial]
fn mid_size_streaming_rejects_inconsistent_part_geometry() {
let (object_info, mut fi) = plain_metadata(256 * 1024);
fi.parts[0].size += 1;
assert_eq!(
get_mid_size_streaming_object_size_with_flags(&None, &object_info, &fi, &ObjectOptions::default(), true, true, true),
None
);
let (mut object_info, fi) = plain_metadata(256 * 1024);
object_info.parts = Arc::new(vec![ObjectPartInfo {
number: 2,
size: fi.parts[0].size,
actual_size: fi.parts[0].actual_size,
..Default::default()
}]);
assert_eq!(
get_mid_size_streaming_object_size_with_flags(&None, &object_info, &fi, &ObjectOptions::default(), true, true, true),
None
);
}
#[test]
#[serial]
fn mid_size_streaming_can_be_disabled_without_affecting_direct_memory_gate() {
temp_env::with_vars(
[
(ENV_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE, Some("false")),
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_ENABLE, Some("true")),
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED, Some("true")),
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED, Some("true")),
],
|| {
let (object_info, fi) = plain_metadata(256 * 1024);
assert_eq!(
get_mid_size_streaming_object_size(&None, &object_info, &fi, &ObjectOptions::default(), true),
None
);
},
);
}
#[test]
#[serial]
fn mid_size_streaming_respects_codec_compatibility_kill_switches() {
let (object_info, fi) = plain_metadata(256 * 1024);
for (name, value) in [
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_ENABLE, "false"),
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED, "false"),
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED, "false"),
] {
temp_env::with_vars(
[
(ENV_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE, Some("true")),
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_ENABLE, Some("true")),
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED, Some("true")),
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED, Some("true")),
(name, Some(value)),
],
|| {
assert_eq!(
get_mid_size_streaming_object_size(&None, &object_info, &fi, &ObjectOptions::default(), true),
None
);
},
);
}
}
#[test]
#[serial]
fn mid_size_streaming_rejects_copy_source_policy() {
let (object_info, fi) = plain_metadata(256 * 1024);
let result = tokio::runtime::Runtime::new()
.expect("test runtime should initialize")
.block_on(crate::set_disk::with_get_object_read_policy(
crate::set_disk::GetObjectReadPolicy::CopySource,
async {
get_mid_size_streaming_object_size_with_flags(
&None,
&object_info,
&fi,
&ObjectOptions::default(),
true,
true,
true,
)
},
));
assert_eq!(result, None);
}
}
#[cfg(test)]
mod object_encryption_resolver_wiring_tests {
use super::*;
@@ -9297,7 +9678,7 @@ mod replication_lww_tests {
mod inline_put_commit_path_tests {
use super::hermetic_set_disks_support::hermetic_set_disks_isolated as hermetic_set_disks;
use super::*;
use crate::config::storageclass::lookup_config_for_pools_without_env;
use crate::config::storageclass::{INLINE_BLOCK_ENV, lookup_config_for_pools, lookup_config_for_pools_without_env};
use crate::disk::ReadOptions;
use rustfs_config::server_config::KVS;
use serial_test::serial;
@@ -9364,6 +9745,62 @@ mod inline_put_commit_path_tests {
assert_eq!(restored, payload);
}
#[tokio::test]
#[serial]
async fn get_object_reader_wires_mid_size_to_single_inflight() {
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "mid-size-reader-wiring";
let object = "object.bin";
let payload: Vec<u8> = (0..256 * 1024).map(|index| (index % 251) as u8).collect();
make_bucket(&disk_stores, bucket).await;
let storage_class = temp_env::with_var(INLINE_BLOCK_ENV, Some("1KiB"), || lookup_config_for_pools(&KVS::new(), &[4]))
.expect("test storage class should resolve");
set_disks.set_test_storage_class_config(storage_class);
let mut writer = PutObjReader::from_vec(payload.clone());
temp_env::async_with_vars(
[
(ENV_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE, Some("true")),
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_ENABLE, Some("true")),
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED, Some("true")),
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED, Some("true")),
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_ROLLOUT, Some("on")),
(rustfs_config::ENV_OBJECT_LOCK_OPTIMIZATION_ENABLE, Some("true")),
],
async {
set_disks
.put_object(bucket, object, &mut writer, &ObjectOptions::default())
.await
.expect("mid-size wiring fixture should commit");
crate::set_disk::reset_test_get_object_reader_path();
let single_inflight_before = crate::set_disk::coding::decode_reader::test_single_inflight_construction_count();
let mut reader = set_disks
.get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
.await
.expect("mid-size wiring GET should succeed");
let mut restored = Vec::new();
reader
.stream
.read_to_end(&mut restored)
.await
.expect("mid-size wiring reader should stream");
assert_eq!(restored, payload);
assert!(
crate::set_disk::test_get_object_reader_selected_mid_size(),
"full get_object_reader path must select mid-size streaming (path id {})",
crate::set_disk::test_get_object_reader_path_id()
);
assert!(
crate::set_disk::coding::decode_reader::test_single_inflight_construction_count() > single_inflight_before,
"mid-size get_object_reader wiring must construct SingleInFlight"
);
},
)
.await;
}
#[tokio::test]
async fn repeated_gets_reuse_the_set_erasure_shell() {
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
@@ -15493,6 +15930,161 @@ mod put_object_tmp_cleanup_tests {
assert_eq!(body, b"new client body");
}
#[tokio::test]
async fn conditional_put_does_not_block_reads_during_body_ingestion() {
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "conditional-put-nonblocking-read";
let object = "object";
for disk in &disk_stores {
disk.make_volume(bucket).await.expect("bucket volume should be created");
}
let mut initial_reader = PutObjReader::from_vec(b"old body".to_vec());
let initial = set_disks
.put_object(bucket, object, &mut initial_reader, &ObjectOptions::default())
.await
.expect("initial object should be written");
let initial_etag = initial.etag.clone().expect("initial object should have an etag");
let body = vec![b'c'; 64 * 1024];
let split = body.len() / 2;
let (mut source, stream) = tokio::io::duplex(64);
let hash_reader = HashReader::from_stream(
stream,
i64::try_from(body.len()).expect("body length should fit i64"),
i64::try_from(body.len()).expect("body length should fit i64"),
None,
None,
false,
)
.expect("conditional hash reader should be created");
let writer_store = Arc::clone(&set_disks);
let etag_for_put = initial_etag.clone();
let put = tokio::spawn(async move {
let mut reader = PutObjReader::new(hash_reader);
writer_store
.put_object(
bucket,
object,
&mut reader,
&ObjectOptions {
http_preconditions: Some(HTTPPreconditions {
if_match: Some(etag_for_put),
..Default::default()
}),
..Default::default()
},
)
.await
});
source
.write_all(&body[..split])
.await
.expect("conditional PUT should consume the first half of the body");
let info = tokio::time::timeout(
Duration::from_secs(5),
set_disks.get_object_info(bucket, object, &ObjectOptions::default()),
)
.await
.expect("reads must not wait for the conditional PUT body")
.expect("the old version must stay readable during body ingestion");
assert_eq!(info.etag.as_deref(), Some(initial_etag.as_str()));
source
.write_all(&body[split..])
.await
.expect("conditional PUT should consume the remaining body");
drop(source);
put.await
.expect("conditional PUT task should join")
.expect("conditional PUT should commit after the body completes");
}
#[tokio::test]
async fn conditional_put_precondition_is_rechecked_at_commit() {
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "conditional-put-commit-recheck";
let object = "object";
for disk in &disk_stores {
disk.make_volume(bucket).await.expect("bucket volume should be created");
}
let mut initial_reader = PutObjReader::from_vec(b"old body".to_vec());
let initial = set_disks
.put_object(bucket, object, &mut initial_reader, &ObjectOptions::default())
.await
.expect("initial object should be written");
let initial_etag = initial.etag.clone().expect("initial object should have an etag");
let body = vec![b'c'; 64 * 1024];
let split = body.len() / 2;
let (mut source, stream) = tokio::io::duplex(64);
let hash_reader = HashReader::from_stream(
stream,
i64::try_from(body.len()).expect("body length should fit i64"),
i64::try_from(body.len()).expect("body length should fit i64"),
None,
None,
false,
)
.expect("conditional hash reader should be created");
let writer_store = Arc::clone(&set_disks);
let put = tokio::spawn(async move {
let mut reader = PutObjReader::new(hash_reader);
writer_store
.put_object(
bucket,
object,
&mut reader,
&ObjectOptions {
http_preconditions: Some(HTTPPreconditions {
if_match: Some(initial_etag),
..Default::default()
}),
..Default::default()
},
)
.await
});
source
.write_all(&body[..split])
.await
.expect("conditional PUT should consume the first half of the body");
let mut interloper_reader = PutObjReader::from_vec(b"interloper body".to_vec());
tokio::time::timeout(
Duration::from_secs(5),
set_disks.put_object(bucket, object, &mut interloper_reader, &ObjectOptions::default()),
)
.await
.expect("the interloper write must not wait for the conditional PUT body")
.expect("the interloper write should commit while the conditional PUT streams");
source
.write_all(&body[split..])
.await
.expect("conditional PUT should consume the remaining body");
drop(source);
let err = put
.await
.expect("conditional PUT task should join")
.expect_err("the conditional PUT must recheck its precondition under the commit lock");
assert_eq!(err, StorageError::PreconditionFailed);
let mut reader = set_disks
.get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
.await
.expect("the interloper object should remain readable");
let mut body = Vec::new();
reader
.stream
.read_to_end(&mut body)
.await
.expect("the interloper object should drain");
assert_eq!(body, b"interloper body");
}
#[tokio::test]
async fn metadata_copy_no_lock_aborts_after_outer_namespace_lock_loss() {
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
+282 -16
View File
@@ -52,6 +52,8 @@ use std::{
};
use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
pub(super) const GET_OBJECT_PATH_MID_SIZE_STREAMING: &str = "mid_size_streaming";
#[cfg(test)]
use super::DEFAULT_GET_OBJECT_METADATA_CACHE_MAX_ENTRIES;
#[cfg(test)]
@@ -1340,6 +1342,80 @@ impl SetDisks {
#[allow(clippy::too_many_arguments)]
#[hotpath::measure(impl_type = "SetDisks")]
pub(super) async fn get_object_decode_reader_with_fileinfo(
bucket: &str,
object: &str,
erasure_cache: Arc<ErasureCache>,
fi: &FileInfo,
files: &[FileInfo],
disks: &[Option<DiskStore>],
set_index: usize,
pool_index: usize,
skip_verify_bitrot: bool,
metrics_object_class: &'static str,
metrics_size_bucket: &'static str,
prefer_data_blocks_first_reader_setup: bool,
) -> Result<GetCodecStreamingReaderBuildOutcome> {
Self::get_object_decode_reader_with_fileinfo_inner(
bucket,
object,
erasure_cache,
fi,
files,
disks,
set_index,
pool_index,
skip_verify_bitrot,
metrics_object_class,
metrics_size_bucket,
prefer_data_blocks_first_reader_setup,
get_codec_streaming_metrics_path(),
false,
false,
)
.await
}
/// Build the bounded mid-size reader while allowing a degraded part to
/// reuse the shard readers it just opened for an in-place legacy fallback.
/// This avoids opening every shard twice when a healthy quorum requires
/// reconstruction.
#[allow(clippy::too_many_arguments)]
pub(super) async fn get_object_mid_size_reader_with_fileinfo(
bucket: &str,
object: &str,
erasure_cache: Arc<ErasureCache>,
fi: &FileInfo,
files: &[FileInfo],
disks: &[Option<DiskStore>],
set_index: usize,
pool_index: usize,
skip_verify_bitrot: bool,
metrics_object_class: &'static str,
metrics_size_bucket: &'static str,
prefer_data_blocks_first_reader_setup: bool,
) -> Result<GetCodecStreamingReaderBuildOutcome> {
Self::get_object_decode_reader_with_fileinfo_inner(
bucket,
object,
erasure_cache,
fi,
files,
disks,
set_index,
pool_index,
skip_verify_bitrot,
metrics_object_class,
metrics_size_bucket,
prefer_data_blocks_first_reader_setup,
GET_OBJECT_PATH_MID_SIZE_STREAMING,
true,
true,
)
.await
}
#[allow(clippy::too_many_arguments)]
async fn get_object_decode_reader_with_fileinfo_inner(
bucket: &str,
object: &str,
erasure_cache: Arc<ErasureCache>,
@@ -1352,6 +1428,9 @@ impl SetDisks {
metrics_object_class: &'static str,
metrics_size_bucket: &'static str,
prefer_data_blocks_first_reader_setup: bool,
metrics_path: &'static str,
allow_inplace_legacy_fallback: bool,
single_inflight: bool,
) -> Result<GetCodecStreamingReaderBuildOutcome> {
let erasure = erasure_cache.get_for_file_info(fi)?;
let (disks, files) = Self::shuffle_disks_and_parts_metadata_by_index(disks, files, fi);
@@ -1375,10 +1454,12 @@ impl SetDisks {
metrics_object_class,
metrics_size_bucket,
prefer_data_blocks_first_reader_setup,
single_inflight,
// Single-part objects keep the whole-request fallback: a degraded
// sole part is detected before any byte streams, so the caller can
// still hand the request to the legacy duplex path unchanged.
false,
allow_inplace_legacy_fallback,
metrics_path,
)
.await;
}
@@ -1427,10 +1508,12 @@ impl SetDisks {
metrics_object_class,
metrics_size_bucket,
false,
false,
// The first part stays eager and keeps the whole-request fallback:
// if part 1 is already degraded, the entire GET drops to the legacy
// duplex path before a single byte is streamed (semantics unchanged).
false,
metrics_path,
)
.await?
{
@@ -1452,6 +1535,7 @@ impl SetDisks {
skip_verify_bitrot,
metrics_object_class,
metrics_size_bucket,
metrics_path,
});
let builder: LazyPartBuilder = Box::new(move |remaining_index| {
let ctx = Arc::clone(&ctx);
@@ -1472,18 +1556,20 @@ impl SetDisks {
ctx.metrics_object_class,
ctx.metrics_size_bucket,
false,
false,
// backlog#879: later parts have already streamed earlier bytes,
// so a whole-request fallback is impossible here. Degrade this
// part in place to a legacy per-part decode reader instead of
// failing the stream mid-flight.
true,
ctx.metrics_path,
)
.await
})
});
Ok(GetCodecStreamingReaderBuildOutcome::Reader(Box::new(
LazyMultipartCodecStreamingReader::new(first_reader, total_parts, builder, get_codec_streaming_metrics_path()),
LazyMultipartCodecStreamingReader::new(first_reader, total_parts, builder, metrics_path),
)))
}
@@ -1504,14 +1590,9 @@ impl SetDisks {
metrics_object_class: &'static str,
metrics_size_bucket: &'static str,
prefer_data_blocks_first_reader_setup: bool,
// backlog#879: when the codec streaming fast path cannot serve this part
// (a shard is missing and reconstruction is required), `false` preserves
// the historical whole-request fallback by returning `Fallback`, while
// `true` degrades in place — building a legacy per-part decode reader that
// reuses the shard readers already opened here. Only lazily-built later
// parts pass `true`, so the eager first-part fallback semantics are
// untouched and the common read path is never affected.
single_inflight: bool,
allow_inplace_legacy_fallback: bool,
metrics_path: &'static str,
) -> Result<GetCodecStreamingReaderBuildOutcome> {
if part_length > part_size {
return Err(Error::other("codec streaming reader part length exceeds part size"));
@@ -1528,7 +1609,6 @@ impl SetDisks {
let read_length = till_offset.saturating_sub(read_offset);
let stage_metrics_enabled = rustfs_io_metrics::get_stage_metrics_enabled();
let metrics_path = get_codec_streaming_metrics_path();
let reader_stage_metrics = stage_metrics_enabled.then_some(BitrotReaderStageMetrics {
path: metrics_path,
reader_construction_stage: GET_STAGE_READER_TASK_READER_CONSTRUCTION,
@@ -1631,11 +1711,23 @@ impl SetDisks {
.with_deferred_parity_handles(deferred_stripe_handles)
.with_deferred_parity_reopeners(deferred_reopeners);
let engine = build_get_codec_streaming_decode_engine(erasure.clone())?;
let reader =
coding::decode_reader::ErasureDecodeReader::new_with_metrics_path(source, engine, part_length, metrics_path)?;
Ok(GetCodecStreamingReaderBuildOutcome::Reader(Box::new(
coding::decode_reader::SyncErasureDecodeReader::new_with_metrics_path(reader, metrics_path),
)))
let reader = if single_inflight {
coding::decode_reader::ErasureDecodeReader::new_single_inflight_with_metrics_path(
source,
engine,
part_length,
metrics_path,
)?
} else {
coding::decode_reader::ErasureDecodeReader::new_with_metrics_path(source, engine, part_length, metrics_path)?
};
if single_inflight {
Ok(GetCodecStreamingReaderBuildOutcome::Reader(Box::new(reader)))
} else {
Ok(GetCodecStreamingReaderBuildOutcome::Reader(Box::new(
coding::decode_reader::SyncErasureDecodeReader::new_with_metrics_path(reader, metrics_path),
)))
}
}
}
@@ -1751,6 +1843,7 @@ struct LazyCodecPartContext {
skip_verify_bitrot: bool,
metrics_object_class: &'static str,
metrics_size_bucket: &'static str,
metrics_path: &'static str,
}
type LazyPartBuildHandle = tokio::task::JoinHandle<Result<GetCodecStreamingReaderBuildOutcome>>;
@@ -4594,6 +4687,154 @@ mod tests {
assert_eq!(body, part_data);
}
#[tokio::test]
#[serial_test::serial]
async fn mid_size_reader_restores_plain_objects_without_duplex() {
for size in [256_usize * 1024, 1024_usize * 1024] {
let payload = (0..size)
.map(|index| u8::try_from(index % 251).expect("pattern byte fits u8"))
.collect::<Vec<_>>();
let erasure = coding::Erasure::new(4, 2, 1024 * 1024);
let mut fi = codec_streaming_test_fileinfo(i64::try_from(size).expect("test size fits i64"), 1);
fi.erasure.block_size = erasure.block_size;
fi.erasure.distribution = (1..=erasure.total_shard_count()).collect();
let files = codec_streaming_inline_files(&erasure, &payload).await;
let (_dirs, disks) = local_test_disks(files.len(), CODEC_STREAMING_TEST_BUCKET).await;
let outcome = SetDisks::get_object_mid_size_reader_with_fileinfo(
CODEC_STREAMING_TEST_BUCKET,
CODEC_STREAMING_TEST_OBJECT,
Arc::new(ErasureCache::new()),
&fi,
&files,
&disks,
0,
0,
false,
"plain_single_part",
"le_1mib",
false,
)
.await
.expect("mid-size reader setup should succeed");
let GetCodecStreamingReaderBuildOutcome::Reader(mut reader) = outcome else {
panic!("mid-size plain object should use streaming reader");
};
let mut body = Vec::new();
reader
.read_to_end(&mut body)
.await
.expect("mid-size reader should restore full body");
assert_eq!(body, payload, "mid-size reader must preserve every payload byte for object size {size}");
}
}
#[tokio::test]
#[serial_test::serial]
async fn mid_size_reader_restores_plain_objects_from_external_part_files() {
for size in [256_usize * 1024, 1024_usize * 1024] {
let payload = (0..size)
.map(|index| u8::try_from(index % 251).expect("pattern byte fits u8"))
.collect::<Vec<_>>();
let erasure = coding::Erasure::new(4, 2, 1024 * 1024);
let mut fi = codec_streaming_test_fileinfo(i64::try_from(size).expect("test size fits i64"), 1);
fi.erasure.block_size = erasure.block_size;
fi.erasure.distribution = (1..=erasure.total_shard_count()).collect();
let (_dirs, disks) = local_test_disks(erasure.total_shard_count(), CODEC_STREAMING_TEST_BUCKET).await;
let (data_dir, files) = codec_streaming_external_files(&erasure, &payload, &disks).await;
fi.data_dir = Some(data_dir);
let outcome = SetDisks::get_object_mid_size_reader_with_fileinfo(
CODEC_STREAMING_TEST_BUCKET,
CODEC_STREAMING_TEST_OBJECT,
Arc::new(ErasureCache::new()),
&fi,
&files,
&disks,
0,
0,
false,
"plain_single_part",
"le_1mib",
false,
)
.await
.expect("external mid-size reader setup should succeed");
let GetCodecStreamingReaderBuildOutcome::Reader(mut reader) = outcome else {
panic!("external plain object should use streaming reader");
};
let mut body = Vec::new();
reader
.read_to_end(&mut body)
.await
.expect("external mid-size reader should restore full body");
assert_eq!(
body, payload,
"external part files must preserve every payload byte for object size {size}"
);
}
}
#[test]
fn minio_large_object_metadata_remains_reader_compatible() {
let raw = rustfs_filemeta::test_data::create_minio_large_object_xlmeta().expect("load MinIO large-object fixture");
let file_info = rustfs_filemeta::FileMeta::load(&raw)
.expect("MinIO xl.meta should decode")
.into_fileinfo("interop", "large.bin", "", false, false, true)
.expect("MinIO xl.meta should materialize FileInfo");
file_info
.validate_for_metadata_read()
.expect("MinIO metadata should satisfy the reader validation contract");
assert_eq!(file_info.size, 300_000);
assert_eq!(file_info.parts.len(), 1);
assert!(file_info.data_dir.is_some());
assert!(!file_info.inline_data());
}
#[tokio::test]
#[serial_test::serial]
async fn mid_size_reader_reuses_open_readers_for_degraded_quorum() {
let size = 256 * 1024;
let payload = vec![0x5a; size];
let erasure = coding::Erasure::new(4, 2, 1024 * 1024);
let mut fi = codec_streaming_test_fileinfo(i64::try_from(size).expect("test size fits i64"), 1);
fi.erasure.block_size = erasure.block_size;
fi.erasure.distribution = (1..=erasure.total_shard_count()).collect();
let mut files = codec_streaming_inline_files(&erasure, &payload).await;
files[0].data = None;
let (_dirs, disks) = local_test_disks(files.len(), CODEC_STREAMING_TEST_BUCKET).await;
let outcome = SetDisks::get_object_mid_size_reader_with_fileinfo(
CODEC_STREAMING_TEST_BUCKET,
CODEC_STREAMING_TEST_OBJECT,
Arc::new(ErasureCache::new()),
&fi,
&files,
&disks,
0,
0,
false,
"plain_single_part",
"le_1mib",
false,
)
.await
.expect("degraded mid-size reader setup should retain read quorum");
let GetCodecStreamingReaderBuildOutcome::Reader(mut reader) = outcome else {
panic!("degraded quorum should use in-place legacy fallback reader");
};
let mut body = Vec::new();
reader
.read_to_end(&mut body)
.await
.expect("degraded reader should reconstruct full body");
assert_eq!(body, payload);
}
#[tokio::test]
async fn get_object_with_fileinfo_restores_missing_inline_data_shard_and_submits_repair() {
let part_data = b"abcdefgh";
@@ -4654,6 +4895,8 @@ mod tests {
"test-size-bucket",
false,
false,
false,
get_codec_streaming_metrics_path(),
)
.await;
assert!(oversized.is_err(), "part_length > part_size must be rejected");
@@ -4674,6 +4917,8 @@ mod tests {
"test-size-bucket",
false,
false,
false,
get_codec_streaming_metrics_path(),
)
.await;
assert!(missing_quorum.is_err(), "reader setup must fail closed when no shard can answer");
@@ -5208,7 +5453,7 @@ mod tests {
Ok(decoded)
}
async fn codec_streaming_inline_files(erasure: &coding::Erasure, part_data: &'static [u8]) -> Vec<FileInfo> {
async fn codec_streaming_inline_files(erasure: &coding::Erasure, part_data: &[u8]) -> Vec<FileInfo> {
let shards = erasure.encode_data(part_data).expect("test part should encode");
let distribution = (1..=erasure.total_shard_count()).collect::<Vec<_>>();
let mut files = Vec::with_capacity(shards.len());
@@ -5231,6 +5476,27 @@ mod tests {
files
}
async fn codec_streaming_external_files(
erasure: &coding::Erasure,
part_data: &[u8],
disks: &[Option<crate::disk::DiskStore>],
) -> (Uuid, Vec<FileInfo>) {
let data_dir = Uuid::from_u128(0x2060_0000_0000_0000_0000_0000_0000_0001);
let mut files = codec_streaming_inline_files(erasure, part_data).await;
for (index, file) in files.iter_mut().enumerate() {
let shard = file.data.take().expect("external fixture shard should be encoded");
file.data_dir = Some(data_dir);
let path = format!("{CODEC_STREAMING_TEST_OBJECT}/{data_dir}/part.1");
disks[index]
.as_ref()
.expect("external fixture disk should be online")
.write_all(CODEC_STREAMING_TEST_BUCKET, &path, shard)
.await
.expect("external fixture shard should be written");
}
(data_dir, files)
}
#[tokio::test]
async fn bitrot_reader_setup_stops_at_read_quorum() {
let setup = setup_inline_bitrot_readers(
+196 -16
View File
@@ -99,6 +99,8 @@ fn preflight_startup_rpc_secret_with(
const LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES: usize = 6;
const LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY: Duration = Duration::from_secs(60 * 3);
const LOCAL_DECOMMISSION_RESUME_RETRY_DELAY: Duration = Duration::from_secs(30);
const REBALANCE_INITIAL_RESUME_DELAY: Duration = Duration::from_secs(10);
const REBALANCE_RESUME_RETRY_DELAY: Duration = Duration::from_secs(10);
fn should_retry_local_decommission_resume(err: &Error, attempt: usize) -> bool {
matches!(err, Error::ConfigNotFound) && attempt < LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES
@@ -108,8 +110,12 @@ fn should_retry_format_load(err: &Error) -> bool {
!matches!(err, Error::CorruptedFormat)
}
fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_meta_loaded: bool) -> bool {
rebalance_meta_loaded && !decommission_running
fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_resume_required: bool) -> bool {
rebalance_resume_required && !decommission_running
}
fn should_defer_rebalance_auto_start(distributed: bool, fleet_proof_available: bool) -> bool {
distributed && !fleet_proof_available
}
fn should_schedule_local_decommission_resume(
@@ -127,6 +133,17 @@ async fn wait_for_local_decommission_resume_delay(rx: &CancellationToken, delay:
}
}
async fn wait_for_rebalance_resume_delay(rx: &CancellationToken, delay: Duration) -> bool {
tokio::select! {
_ = rx.cancelled() => false,
_ = tokio::time::sleep(delay) => true,
}
}
async fn wait_for_rebalance_resume_retry(rx: &CancellationToken) -> bool {
wait_for_rebalance_resume_delay(rx, REBALANCE_RESUME_RETRY_DELAY).await
}
fn resolve_store_init_stage_result(result: Result<()>, stage: &str) -> Result<()> {
result.map_err(|err| Error::other(format!("store init failed during {stage}: {err}")))
}
@@ -283,6 +300,71 @@ async fn resume_local_decommission_after_init(store: Arc<ECStore>, rx: Cancellat
}
}
async fn resume_rebalance_after_init(store: Arc<ECStore>, rx: CancellationToken) {
if !wait_for_rebalance_resume_delay(&rx, REBALANCE_INITIAL_RESUME_DELAY).await {
return;
}
loop {
if rx.is_cancelled() {
return;
}
let resume_required = store
.rebalance_meta
.read()
.await
.as_ref()
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation);
if !resume_required {
return;
}
if should_defer_rebalance_auto_start(
store.ctx.is_dist_erasure().await,
crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_some(),
) {
if !wait_for_rebalance_resume_retry(&rx).await {
return;
}
continue;
}
match store.start_rebalance().await {
Ok(()) => return,
Err(err) if crate::core::pools::is_pool_activation_fleet_proof_error(&err) => {
warn!(
event = EVENT_ECSTORE_INIT_STATUS,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_STORE_INIT,
stage = "start_rebalance",
state = "retrying",
reason = "fleet_capability_proof_unavailable",
error = %err,
retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
"Retrying deferred rebalance auto-start"
);
if !wait_for_rebalance_resume_retry(&rx).await {
return;
}
}
Err(err) => {
error!(
event = EVENT_ECSTORE_INIT_STATUS,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_STORE_INIT,
stage = "start_rebalance",
state = "failed",
reason = "deferred_resume_failed",
error = %err,
"Failed to resume rebalance after store initialization"
);
return;
}
}
}
}
impl ECStore {
/// Validate topology and process storage-class overrides before any disk is opened.
pub fn validate_startup_storage_class(endpoint_pools: &EndpointServerPools) -> Result<()> {
@@ -574,12 +656,49 @@ impl ECStore {
}
resolve_store_init_stage_result(self.load_rebalance_meta().await, "load_rebalance_meta")?;
let rebalance_meta_loaded = self.rebalance_meta.read().await.is_some();
let rebalance_resume_required = {
let rebalance_meta = self.rebalance_meta.read().await;
rebalance_meta
.as_ref()
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
};
let decommission_running =
pool_meta_has_active_decommission(&installed_pool_meta) || self.is_decommission_running().await;
if should_auto_start_rebalance_after_init(decommission_running, rebalance_meta_loaded) {
resolve_store_init_stage_result(self.start_rebalance().await, "start_rebalance")?;
} else if decommission_running && rebalance_meta_loaded {
let distributed = self.ctx.is_dist_erasure().await;
let fleet_proof_available = crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_some();
let mut rebalance_auto_start_deferred = false;
if should_auto_start_rebalance_after_init(decommission_running, rebalance_resume_required) {
if should_defer_rebalance_auto_start(distributed, fleet_proof_available) {
rebalance_auto_start_deferred = true;
warn!(
event = EVENT_ECSTORE_INIT_STATUS,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_STORE_INIT,
stage = "start_rebalance",
state = "deferred",
reason = "fleet_capability_proof_unavailable",
retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
"Deferred rebalance auto-start until a live fleet capability proof is available"
);
} else if let Err(err) = self.start_rebalance().await {
if crate::core::pools::is_pool_activation_fleet_proof_error(&err) {
rebalance_auto_start_deferred = true;
warn!(
event = EVENT_ECSTORE_INIT_STATUS,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_STORE_INIT,
stage = "start_rebalance",
state = "deferred",
reason = "fleet_capability_proof_changed",
error = %err,
retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
"Deferred rebalance auto-start after the fleet capability proof changed"
);
} else {
return resolve_store_init_stage_result(Err(err), "start_rebalance");
}
}
} else if decommission_running && rebalance_resume_required {
warn!(
event = EVENT_ECSTORE_INIT_STATUS,
component = LOG_COMPONENT_ECSTORE,
@@ -616,12 +735,13 @@ impl ECStore {
.is_ok();
if should_schedule_local_decommission_resume(&local_pool_indices, pool_meta_replica_state, pool_meta_write_safe) {
let store = self.clone();
let decommission_rx = rx.clone();
tokio::spawn(async move {
if !wait_for_local_decommission_resume_delay(&rx, LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY).await {
if !wait_for_local_decommission_resume_delay(&decommission_rx, LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY).await {
return;
}
resume_local_decommission_after_init(store, rx, local_pool_indices).await;
resume_local_decommission_after_init(store, decommission_rx, local_pool_indices).await;
});
} else if !local_pool_indices.is_empty() {
error!(
@@ -648,6 +768,11 @@ impl ECStore {
info!("TierConfigMgr init error: {}", err);
}
if rebalance_auto_start_deferred {
let store = self.clone();
tokio::spawn(resume_rebalance_after_init(store, rx));
}
Ok(())
}
@@ -665,7 +790,8 @@ mod tests {
load_pool_meta_for_startup, persist_pool_meta_for_startup_if_safe, pool_first_endpoint_is_local,
pool_meta_has_active_decommission, preflight_startup_rpc_secret_with, resolve_startup_pool_defaults_with,
resolve_store_init_stage_result, save_validated_pool_meta_for_startup, should_auto_start_rebalance_after_init,
should_retry_format_load, should_retry_local_decommission_resume, wait_for_local_decommission_resume_delay,
should_defer_rebalance_auto_start, should_retry_format_load, should_retry_local_decommission_resume,
wait_for_local_decommission_resume_delay,
};
#[cfg(feature = "test-util")]
use crate::disk::DiskAPI;
@@ -1450,7 +1576,7 @@ mod tests {
}
#[test]
fn test_should_auto_start_rebalance_after_init_allows_loaded_rebalance_without_decommission() {
fn test_should_auto_start_rebalance_after_init_allows_active_rebalance_without_decommission() {
assert!(should_auto_start_rebalance_after_init(false, true));
}
@@ -1460,10 +1586,17 @@ mod tests {
}
#[test]
fn test_should_auto_start_rebalance_after_init_rejects_missing_rebalance_meta() {
fn test_should_auto_start_rebalance_after_init_rejects_terminal_or_missing_rebalance() {
assert!(!should_auto_start_rebalance_after_init(false, false));
}
#[test]
fn test_should_defer_rebalance_auto_start_only_without_distributed_fleet_proof() {
assert!(should_defer_rebalance_auto_start(true, false));
assert!(!should_defer_rebalance_auto_start(true, true));
assert!(!should_defer_rebalance_auto_start(false, false));
}
#[test]
fn test_store_init_recovery_skips_rebalance_when_decommission_metadata_is_active() {
let pool_meta = init_test_pool_meta(Some(PoolDecommissionInfo {
@@ -1473,22 +1606,69 @@ mod tests {
canceled: false,
..Default::default()
}));
let rebalance_meta = Some(RebalanceMeta::default());
let rebalance_meta = Some(RebalanceMeta {
pool_stats: vec![RebalanceStats {
participating: true,
info: RebalanceInfo {
status: RebalStatus::Started,
..Default::default()
},
..Default::default()
}],
..Default::default()
});
assert!(!should_auto_start_rebalance_after_init(
pool_meta_has_active_decommission(&pool_meta),
rebalance_meta.is_some()
rebalance_meta
.as_ref()
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
));
}
#[test]
fn test_store_init_recovery_allows_rebalance_when_only_rebalance_metadata_exists() {
fn test_store_init_recovery_allows_active_rebalance_without_decommission() {
let pool_meta = init_test_pool_meta(None);
let rebalance_meta = Some(RebalanceMeta::default());
let rebalance_meta = Some(RebalanceMeta {
pool_stats: vec![RebalanceStats {
participating: true,
info: RebalanceInfo {
status: RebalStatus::Started,
..Default::default()
},
..Default::default()
}],
..Default::default()
});
assert!(should_auto_start_rebalance_after_init(
pool_meta_has_active_decommission(&pool_meta),
rebalance_meta.is_some()
rebalance_meta
.as_ref()
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
));
}
#[test]
fn test_store_init_recovery_skips_completed_rebalance_metadata() {
let pool_meta = init_test_pool_meta(None);
let rebalance_meta = Some(RebalanceMeta {
pool_stats: vec![RebalanceStats {
participating: true,
info: RebalanceInfo {
status: RebalStatus::Completed,
..Default::default()
},
..Default::default()
}],
..Default::default()
});
assert!(!should_auto_start_rebalance_after_init(
pool_meta_has_active_decommission(&pool_meta),
rebalance_meta
.as_ref()
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
));
}
+35 -1
View File
@@ -2110,9 +2110,18 @@ fn build_list_versions_next_marker(
cache_id: Option<&str>,
) -> (Option<String>, Option<String>) {
if let Some(last) = objects.last() {
// A null version carries the synthesized `Some(Uuid::nil())` identity
// here; advertise it as the literal `null` marker so a resumed listing
// parses it back to `VersionMarker::Null` instead of a nil UUID that
// `find_version_index` can never match (issue #6745).
(
Some(append_list_cache_id_to_marker(last.name.clone(), cache_id)),
Some(last.version_id.map(|v| v.to_string()).unwrap_or_else(|| "null".to_string())),
Some(
last.version_id
.filter(|v| !v.is_nil())
.map(|v| v.to_string())
.unwrap_or_else(|| "null".to_string()),
),
)
} else if let Some(last_prefix) = prefixes.last() {
(Some(append_list_cache_id_to_marker(last_prefix.clone(), cache_id)), None)
@@ -7759,6 +7768,31 @@ mod test {
out
}
// A truncated versions listing ending on a null version must advertise the
// literal `null` continuation marker: a nil UUID parses to
// `VersionMarker::Version(nil)`, which no stored version matches, so the
// resumed page would replay every version (issue #6745).
#[test]
fn build_list_versions_next_marker_reports_null_for_nil_version_id() {
let null_version = ObjectInfo {
name: "obj-a".to_owned(),
version_id: Some(uuid::Uuid::nil()),
..Default::default()
};
let real_version = ObjectInfo {
name: "obj-b".to_owned(),
version_id: Some(uuid::Uuid::from_u128(7)),
..Default::default()
};
let (next_marker, next_version_idmarker) = super::build_list_versions_next_marker(&[null_version], &[], None);
assert_eq!(next_marker.as_deref(), Some("obj-a"));
assert_eq!(next_version_idmarker.as_deref(), Some("null"));
let (_, next_version_idmarker) = super::build_list_versions_next_marker(&[real_version], &[], None);
assert_eq!(next_version_idmarker.as_deref(), Some(uuid::Uuid::from_u128(7).to_string().as_str()));
}
// ECA-03 / #944: a page whose raw keys fully collapse into fewer than max_keys
// common prefixes must still report truncation and carry a continuation marker,
// otherwise every key beyond the scan window is silently dropped.
+20
View File
@@ -16,6 +16,8 @@ use thiserror::Error;
use super::heal::{DiskError, EcstoreError};
const HEAL_DANGLING_DELETE_GRACE_MESSAGE: &str = "dangling object deletion deferred by heal grace window";
/// Custom error type for heal operations
/// This enum defines various error variants that can occur during
/// the execution of heal-related tasks, such as I/O errors, storage errors,
@@ -98,6 +100,9 @@ impl Error {
// them.
Error::Storage(EcstoreError::Lock(lock_err)) => !lock_err.is_fatal(),
Error::Storage(err) => {
if err.is_dangling_delete_grace() {
return true;
}
err.is_quorum_error()
|| matches!(
err,
@@ -110,6 +115,9 @@ impl Error {
|| is_recoverable_heal_error_message(&err.to_string())
}
Error::Disk(err) => {
if err.is_dangling_delete_grace() {
return true;
}
matches!(
err,
DiskError::DiskNotFound
@@ -127,6 +135,18 @@ impl Error {
_ => false,
}
}
pub(crate) fn is_dangling_delete_grace(&self) -> bool {
match self {
Error::Storage(err) => err.is_dangling_delete_grace(),
Error::Disk(err) => err.is_dangling_delete_grace(),
Error::Io(err) => DiskError::io_error_is_dangling_delete_grace(err),
Error::TaskExecutionFailed { message } | Error::Other(message) => {
message.contains(HEAL_DANGLING_DELETE_GRACE_MESSAGE)
}
_ => false,
}
}
}
/// Documented substring fallback for errors that reach heal with their typed
+17 -2
View File
@@ -159,9 +159,13 @@ impl ErasureSetHealer {
/// Classify an error returned by [`HealStorageAPI::heal_object`].
///
/// Both the inner `Ok((_, Some(err)))` and the outer `Err(err)` produced by
/// `heal_object` wrap `Error::Storage(StorageError)`, so match on that.
/// Most heal object failures wrap `Error::Storage(StorageError)`, while
/// compatibility markers can also arrive through Disk/Io/task wrappers.
fn classify_heal_object_error(err: &Error) -> HealObjectOutcome {
if err.is_dangling_delete_grace() {
return HealObjectOutcome::Transient;
}
let Error::Storage(se) = err else {
return HealObjectOutcome::Failed;
};
@@ -1459,6 +1463,7 @@ mod tests {
// genuine object absence, or transient failures get recorded as "healed" and
// permanently skipped.
use super::{EcstoreError, Error, HealObjectOutcome};
use crate::heal::DiskError;
fn classify(err: EcstoreError) -> HealObjectOutcome {
ErasureSetHealer::classify_heal_object_error(&Error::Storage(err))
@@ -1479,6 +1484,16 @@ mod tests {
));
}
#[test]
fn dangling_delete_grace_is_transient() {
assert!(matches!(
ErasureSetHealer::classify_heal_object_error(&Error::Disk(DiskError::other(
"dangling object deletion deferred by heal grace window; retry_after_secs=3599; grace_secs=3600"
))),
HealObjectOutcome::Transient
));
}
#[test]
fn genuine_object_absence_is_absent() {
assert!(matches!(classify(EcstoreError::FileNotFound), HealObjectOutcome::Absent));
+9 -35
View File
@@ -20,7 +20,10 @@ use crate::heal::{
};
use crate::{Error, Result};
use metrics::{counter, gauge};
use rustfs_concurrency::{AdmissionState, WorkloadAdmissionSnapshotProvider, WorkloadClass};
use rustfs_concurrency::WorkloadAdmissionSnapshotProvider;
use rustfs_concurrency::workload::{ForegroundPressure, foreground_pressure};
#[cfg(test)]
use rustfs_concurrency::{AdmissionState, WorkloadClass};
use rustfs_heal_contracts::heal_channel::{
HealAdmissionDropReason, HealAdmissionReceipt, HealAdmissionResult, HealRequestSource,
};
@@ -813,40 +816,11 @@ impl HealManager {
}
let provider = provider.as_ref()?;
let snapshot = provider.workload_admission_snapshot();
[
(WorkloadClass::ForegroundRead, config.mainline_read_utilization_high_percent),
(WorkloadClass::ForegroundWrite, config.mainline_write_utilization_high_percent),
]
.into_iter()
.filter_map(|(class, threshold_pct)| {
if threshold_pct == 0 {
return None;
}
let entry = snapshot.get(class)?;
let usage_pct = if matches!(entry.state, AdmissionState::Saturated) {
100
} else {
let limit = entry.limit?;
if limit == 0 {
return None;
}
entry
.active
.unwrap_or(0)
.saturating_mul(100)
.checked_div(limit)
.unwrap_or(100)
};
(usage_pct >= threshold_pct).then_some(ForegroundPressure {
class,
usage_pct,
threshold_pct,
})
})
.max_by_key(|pressure| pressure.usage_pct)
foreground_pressure(
&provider.workload_admission_snapshot(),
config.mainline_read_utilization_high_percent,
config.mainline_write_utilization_high_percent,
)
}
fn schedule_mainline_throttle_recheck(notify: Arc<Notify>, delay: Duration) {
-17
View File
@@ -78,23 +78,6 @@ pub(super) enum QueuePushOutcome {
Merged,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(super) struct ForegroundPressure {
pub(super) class: WorkloadClass,
pub(super) usage_pct: usize,
pub(super) threshold_pct: usize,
}
impl ForegroundPressure {
pub(super) const fn reason(self) -> &'static str {
match self.class {
WorkloadClass::ForegroundRead => "foreground_read_pressure",
WorkloadClass::ForegroundWrite => "foreground_write_pressure",
_ => "foreground_pressure",
}
}
}
#[derive(Debug, Clone)]
pub(super) struct CompletedHealStatus {
pub(super) heal_type: HealType,
+28
View File
@@ -682,6 +682,10 @@ impl HealTask {
Self::is_data_usage_cache_object(bucket, object) && Self::is_transient_lock_or_timeout_error(err)
}
fn is_dangling_delete_grace_error(err: &Error) -> bool {
err.is_dangling_delete_grace()
}
fn is_no_heal_required_error(err: &Error) -> bool {
match err {
Error::Storage(EcstoreError::NoHealRequired) | Error::Disk(DiskError::NoHealRequired) => true,
@@ -746,6 +750,30 @@ impl HealTask {
true
}
async fn skip_dangling_delete_grace_error(&self, bucket: &str, object: &str, err: &Error) -> bool {
if !Self::is_dangling_delete_grace_error(err) {
return false;
}
warn!(
target: "rustfs::heal::task",
event = EVENT_HEAL_OBJECT_RESULT,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_OBJECT,
task_id = %self.id,
bucket,
object,
result = "dangling_delete_grace_skip",
error = %err,
"Heal object dangling cleanup deferred by grace window"
);
let mut progress = self.progress.write().await;
progress.set_current_object(Some(format!("skipped: {bucket}/{object}")));
progress.update_object_progress(1, 0, 0, 1, 0);
progress.update_stage(3, 3);
true
}
async fn skip_scanner_synthetic_object_dir_missing(&self, bucket: &str, object: &str, err: &Error) -> bool {
if self.source != HealRequestSource::Scanner || !is_missing_object_dir_heal_result(object, err) {
return false;
+15 -1
View File
@@ -359,7 +359,21 @@ impl HealTask {
};
if let Some(err) = error {
if Self::should_skip_data_usage_cache_heal_error(bucket, object, &err) {
if Self::is_dangling_delete_grace_error(&err) {
telemetry_unknown |= !increment_counter(&mut skipped);
warn!(
target: "rustfs::heal::task",
event = EVENT_HEAL_BUCKET_RESULT,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_TASK,
task_id = %self.id,
bucket,
object,
result = "dangling_delete_grace_skip",
error = %err,
"Heal bucket object dangling cleanup deferred by grace window"
);
} else if Self::should_skip_data_usage_cache_heal_error(bucket, object, &err) {
telemetry_unknown |= !increment_counter(&mut skipped);
warn!(
target: "rustfs::heal::task",
+8
View File
@@ -169,6 +169,10 @@ impl HealTask {
match heal_result {
Ok((result, error)) => {
if let Some(e) = error {
if self.skip_dangling_delete_grace_error(bucket, object, &e).await {
return Ok(());
}
if self.skip_data_usage_cache_heal_error(bucket, object, &e).await {
return Ok(());
}
@@ -257,6 +261,10 @@ impl HealTask {
Err(Error::TaskCancelled) => Err(Error::TaskCancelled),
Err(Error::TaskTimeout) => Err(Error::TaskTimeout),
Err(e) => {
if self.skip_dangling_delete_grace_error(bucket, object, &e).await {
return Ok(());
}
if self.skip_data_usage_cache_heal_error(bucket, object, &e).await {
return Ok(());
}
+75
View File
@@ -705,6 +705,7 @@ fn replacement_identity(
enum MockHealObjectOutcome {
OkWithOtherError(&'static str),
ErrOther(&'static str),
DanglingGraceDeferred,
RetryableReadQuorum,
RetryableSlowDown,
PermanentOther(&'static str),
@@ -806,6 +807,12 @@ impl HealStorageAPI for MockStorage {
.and_then(VecDeque::pop_front)
{
return match outcome {
MockHealObjectOutcome::DanglingGraceDeferred => Ok((
HealResultItem::default(),
Some(Error::Disk(DiskError::other(
"dangling object deletion deferred by heal grace window; retry_after_secs=3599; grace_secs=3600",
))),
)),
MockHealObjectOutcome::RetryableReadQuorum => Err(Error::Storage(EcstoreError::InsufficientReadQuorum(
bucket.to_string(),
object.to_string(),
@@ -820,6 +827,12 @@ impl HealStorageAPI for MockStorage {
}
if let Some(outcome) = self.heal_object_outcome.lock().unwrap().take() {
return match outcome {
MockHealObjectOutcome::DanglingGraceDeferred => Ok((
HealResultItem::default(),
Some(Error::Disk(DiskError::other(
"dangling object deletion deferred by heal grace window; retry_after_secs=3599; grace_secs=3600",
))),
)),
MockHealObjectOutcome::OkWithOtherError(message) => Ok((HealResultItem::default(), Some(Error::other(message)))),
MockHealObjectOutcome::ErrOther(message) | MockHealObjectOutcome::PermanentOther(message) => {
Err(Error::other(message))
@@ -1310,6 +1323,31 @@ async fn test_cluster_heal_visits_bucket_objects() {
assert!(matches!(task.get_status().await, HealTaskStatus::Completed));
}
#[tokio::test]
async fn object_heal_skips_dangling_delete_grace_without_failing_task() {
let storage = Arc::new(MockStorage {
heal_object_outcome: Mutex::new(Some(MockHealObjectOutcome::DanglingGraceDeferred)),
..Default::default()
});
let task = HealTask::from_request(
HealRequest::object("bucket-a".to_string(), "recent.txt".to_string(), None),
storage.clone(),
);
task.execute()
.await
.expect("grace-protected dangling cleanup should be reported as a skipped object");
assert!(matches!(task.get_status().await, HealTaskStatus::Completed));
assert!(storage.healed_objects.lock().unwrap().is_empty());
let progress = task.get_progress().await;
assert_eq!(progress.current_object.as_deref(), Some("skipped: bucket-a/recent.txt"));
assert_eq!(progress.objects_scanned, 1);
assert_eq!(progress.objects_healed, 0);
assert_eq!(progress.objects_failed, 0);
assert_eq!(progress.skipped_objects, 1);
}
#[tokio::test(start_paused = true)]
async fn test_recursive_bucket_heal_retries_only_retryable_objects() {
let storage = Arc::new(MockStorage::default());
@@ -1345,6 +1383,43 @@ async fn test_recursive_bucket_heal_retries_only_retryable_objects() {
assert_eq!(progress.objects_failed, 0);
}
#[tokio::test(start_paused = true)]
async fn recursive_bucket_heal_skips_dangling_delete_grace_without_batch_failure() {
let storage = Arc::new(MockStorage::default());
storage
.heal_object_outcomes
.lock()
.unwrap()
.insert("object-a".to_string(), VecDeque::from([MockHealObjectOutcome::DanglingGraceDeferred]));
let request = HealRequest::new(
HealType::Bucket {
bucket: "bucket-a".to_string(),
},
HealOptions {
recursive: true,
timeout: None,
..Default::default()
},
HealPriority::Normal,
);
let task = HealTask::from_request(request, storage.clone());
task.heal_bucket("bucket-a")
.await
.expect("grace-protected dangling cleanup should not fail the bucket heal batch");
assert_eq!(
storage.heal_object_calls.lock().unwrap().as_slice(),
["object-a".to_string(), "object-b".to_string()]
);
assert_eq!(storage.healed_objects.lock().unwrap().as_slice(), ["object-b".to_string()]);
let progress = task.get_progress().await;
assert_eq!(progress.objects_scanned, 2);
assert_eq!(progress.objects_healed, 1);
assert_eq!(progress.objects_failed, 0);
assert_eq!(progress.skipped_objects, 1);
}
#[tokio::test(start_paused = true)]
async fn test_recursive_bucket_heal_reports_typed_exhausted_and_permanent_failures() {
let storage = Arc::new(MockStorage::default());
+45 -1
View File
@@ -2320,9 +2320,13 @@ fn filter_policies_from_docs(policy_docs: &CacheEntity<PolicyDoc>, policy_name:
}
fn build_group_desc(name: &str, group_info: GroupInfo, mapped_policy: Option<MappedPolicy>) -> GroupDesc {
// A group without a mapped policy must report the group's own stored
// timestamp, not the wall clock: this value feeds content-addressed
// consumers (the site-replication repair plan hashes it), and a fresh
// clock on every read makes equal states hash differently.
let (policy, updated_at) = mapped_policy
.map(|policy| (policy.policies, Some(policy.update_at)))
.unwrap_or_else(|| (String::new(), Some(OffsetDateTime::now_utc())));
.unwrap_or_else(|| (String::new(), group_info.update_at));
GroupDesc {
name: name.to_string(),
@@ -2347,6 +2351,46 @@ mod tests {
};
use tokio::sync::Notify;
#[test]
fn test_build_group_desc_without_mapping_reports_the_stored_group_timestamp() {
let stamp = OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("valid timestamp");
let group_info = GroupInfo {
version: 1,
status: "enabled".to_string(),
members: vec!["alice".to_string()],
update_at: Some(stamp),
};
let first = build_group_desc("team", group_info.clone(), None);
let second = build_group_desc("team", group_info, None);
assert_eq!(first.updated_at, Some(stamp));
assert_eq!(first.updated_at, second.updated_at);
assert!(first.policy.is_empty());
}
#[test]
fn test_build_group_desc_with_mapping_reports_the_mapping_timestamp() {
let group_stamp = OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("valid timestamp");
let mapping_stamp = OffsetDateTime::from_unix_timestamp(1_700_000_500).expect("valid timestamp");
let group_info = GroupInfo {
version: 1,
status: "enabled".to_string(),
members: vec![],
update_at: Some(group_stamp),
};
let mapping = MappedPolicy {
version: 1,
policies: "readwrite".to_string(),
update_at: mapping_stamp,
};
let desc = build_group_desc("team", group_info, Some(mapping));
assert_eq!(desc.updated_at, Some(mapping_stamp));
assert_eq!(desc.policy, "readwrite");
}
#[derive(Clone)]
struct FailingInitialLoadStore;
+1 -1
View File
@@ -66,7 +66,7 @@ uuid = { workspace = true, features = ["v4", "serde", "fast-rng", "macro-diagnos
[dev-dependencies]
metrics-util = { workspace = true, features = ["debugging"] }
proptest = "1"
serial_test = { workspace = true }
serial_test.workspace = true
temp-env.workspace = true
tokio = { workspace = true, features = ["macros", "fs", "rt-multi-thread"] }
+45
View File
@@ -1568,6 +1568,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn abort_incomplete_multipart_upload_due_accepts_zero_days() {
let initiated = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
@@ -1628,6 +1629,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn predict_expiration_selects_closest_expiry_for_put_object() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
@@ -1874,6 +1876,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn empty_transition_vectors_are_not_active_or_due() {
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
@@ -1939,6 +1942,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn eval_inner_keeps_latest_object_before_days_due() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
@@ -1972,6 +1976,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn eval_inner_transitions_latest_object_after_days_due() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
@@ -2009,6 +2014,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn eval_inner_transitions_latest_object_after_date_due() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let transition_date = base_time - Duration::days(1);
@@ -2048,6 +2054,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn eval_inner_selects_earliest_due_among_multiple_past_due_events() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
// Two enabled rules both yield a past-due DeleteAction and a third yields a
@@ -2161,6 +2168,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn eval_inner_expires_noncurrent_version_after_due() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
@@ -2198,6 +2206,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn eval_inner_skips_noncurrent_expiration_without_successor() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("valid fixed test timestamp");
let lc = BucketLifecycleConfiguration {
@@ -2233,6 +2242,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn eval_inner_missing_successor_does_not_skip_noncurrent_transition() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("valid fixed test timestamp");
let lc = BucketLifecycleConfiguration {
@@ -2275,6 +2285,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn eval_inner_noncurrent_expiration_one_day_respects_due_boundary() {
let successor_time = datetime!(2025-06-15 12:00:00 UTC);
let due = expected_expiry_time(successor_time, 1);
@@ -2316,6 +2327,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn eval_inner_expires_noncurrent_version_immediately_when_zero_days() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
@@ -2353,6 +2365,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn eval_inner_transitions_noncurrent_version_after_due() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
@@ -2428,6 +2441,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn evaluator_honors_newer_noncurrent_versions_retention_count() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = Arc::new(BucketLifecycleConfiguration {
@@ -2716,6 +2730,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn expired_object_delete_marker_ignores_marker_with_noncurrent_versions_present() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
@@ -2792,6 +2807,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn expired_object_delete_marker_deletes_only_delete_marker_immediately() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
@@ -2869,6 +2885,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn expiration_days_deletes_only_expired_delete_marker_when_due() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
@@ -2919,6 +2936,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn expiration_days_uses_earliest_due_rule_for_expired_delete_marker() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let make_rule = |id: &str, days| LifecycleRule {
@@ -3249,6 +3267,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn del_marker_expiration_deletes_marker_and_older_versions_when_due() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("fixed timestamp should be valid");
let lc = BucketLifecycleConfiguration {
@@ -3288,6 +3307,7 @@ mod tests {
// --- TASK-003 tests: Round up to next UTC processing boundary ---
#[test]
#[serial]
fn expected_expiry_time_rounds_up_to_next_midnight_utc() {
with_default_ilm_process_time(|| {
// Object created at 2025-01-15T10:30:45Z, expire in 30 days
@@ -3303,6 +3323,7 @@ mod tests {
}
#[test]
#[serial]
fn expected_expiry_time_immediate_expiry_returns_epoch() {
with_default_ilm_process_time(|| {
let mod_time = datetime!(2025-06-01 12:00:00 UTC);
@@ -3312,6 +3333,7 @@ mod tests {
}
#[test]
#[serial]
fn expected_expiry_time_preserves_exact_midnight_boundary() {
with_default_ilm_process_time(|| {
let mod_time = datetime!(2025-03-01 00:00:00 UTC);
@@ -3321,6 +3343,7 @@ mod tests {
}
#[test]
#[serial]
fn expected_expiry_time_rounds_end_of_day_to_following_midnight() {
with_default_ilm_process_time(|| {
let mod_time = datetime!(2025-06-15 23:59:59 UTC);
@@ -3330,6 +3353,7 @@ mod tests {
}
#[test]
#[serial]
fn expected_expiry_time_uses_canonical_process_time_boundary() {
let mod_time = datetime!(2025-01-15 10:30:45 UTC);
@@ -3342,6 +3366,7 @@ mod tests {
}
#[test]
#[serial]
fn expected_expiry_time_uses_deprecated_process_time_alias() {
let mod_time = datetime!(2025-01-15 10:30:45 UTC);
@@ -3354,6 +3379,7 @@ mod tests {
}
#[test]
#[serial]
fn expected_expiry_time_uses_default_boundary_when_process_time_is_zero_or_invalid() {
let mod_time = datetime!(2025-01-15 10:30:45 UTC);
@@ -3376,6 +3402,7 @@ mod tests {
// (a) Default path (env unset) is byte-identical: one day == 86400s.
#[test]
#[serial]
fn ilm_day_secs_defaults_to_86400_when_unset() {
temp_env::with_var_unset(ENV_ILM_DEBUG_DAY_SECS, || {
assert_eq!(ilm_day_secs(), DEFAULT_ILM_DAY_SECS);
@@ -3404,6 +3431,7 @@ mod tests {
// (b) End-to-end env read scales the day length.
#[test]
#[serial]
fn ilm_day_secs_scales_when_env_set() {
temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("2"), || {
assert_eq!(ilm_day_secs(), 2);
@@ -3412,6 +3440,7 @@ mod tests {
// (c) Invalid env value falls back to 86400.
#[test]
#[serial]
fn ilm_day_secs_falls_back_on_invalid_env() {
temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("bogus"), || {
assert_eq!(ilm_day_secs(), DEFAULT_ILM_DAY_SECS);
@@ -3424,6 +3453,7 @@ mod tests {
// Deadline math scales: with a 1s day and PROCESS_TIME unset, a Days=1 rule is
// due 1s after mod_time (rounded up to the next 1s boundary => same instant).
#[test]
#[serial]
fn expected_expiry_time_scales_with_debug_day_secs() {
let mod_time = datetime!(2025-01-15 10:30:45 UTC);
temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("1"), || {
@@ -3439,6 +3469,7 @@ mod tests {
// days == 0 still yields the immediate-expiry sentinel regardless of the switch.
#[test]
#[serial]
fn expected_expiry_time_zero_days_ignores_debug_day_secs() {
let mod_time = datetime!(2025-06-01 12:00:00 UTC);
temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("2"), || {
@@ -3449,6 +3480,7 @@ mod tests {
// (③) Interaction with an explicit RUSTFS_ILM_PROCESS_TIME: the deadline offset
// uses the accelerated day length, but the rounding boundary honors PROCESS_TIME.
#[test]
#[serial]
fn expected_expiry_time_debug_day_secs_respects_explicit_process_time() {
let mod_time = datetime!(2025-01-15 10:30:00 UTC);
// day == 10s, but round up to the next 60s (PROCESS_TIME) boundary.
@@ -3465,6 +3497,7 @@ mod tests {
// (③) With the switch unset, an explicit PROCESS_TIME behaves exactly as before.
#[test]
#[serial]
fn expected_expiry_time_unset_debug_day_secs_matches_legacy_process_time() {
let mod_time = datetime!(2025-01-15 10:30:45 UTC);
temp_env::with_var_unset(ENV_ILM_DEBUG_DAY_SECS, || {
@@ -3492,6 +3525,7 @@ mod tests {
// The abort-incomplete-multipart deadline path also scales through the switch.
#[test]
#[serial]
fn abort_incomplete_multipart_due_scales_with_debug_day_secs() {
use s3s::dto::AbortIncompleteMultipartUpload;
let initiated = datetime!(2025-01-15 10:30:45 UTC);
@@ -3536,6 +3570,7 @@ mod tests {
// (⑤ evaluator seam) A Days=1 rule fires under RUSTFS_ILM_DEBUG_DAY_SECS=1 once
// `now` advances a few seconds past a mod_time only ~seconds in the past.
#[test]
#[serial]
fn eval_inner_expires_days_one_rule_under_debug_day_secs() {
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
@@ -3584,6 +3619,7 @@ mod tests {
// Absolute Date-based rules must NOT scale with the switch (regression guard).
#[test]
#[serial]
fn eval_inner_date_rule_ignores_debug_day_secs() {
let expiry_date = datetime!(2025-06-01 00:00:00 UTC);
let lc = BucketLifecycleConfiguration {
@@ -3866,6 +3902,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn eval_inner_triggers_delete_all_versions_when_expired_object_all_versions_set() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
@@ -3904,6 +3941,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn expired_object_all_versions_does_not_apply_to_current_delete_marker() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("fixed timestamp should be valid");
let lc = BucketLifecycleConfiguration {
@@ -3933,6 +3971,7 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn eval_inner_uses_delete_action_when_all_versions_not_set() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
@@ -4051,6 +4090,7 @@ mod tests {
use super::*;
use proptest::prelude::*;
use s3s::dto::{NoncurrentVersionExpiration, Tag};
use serial_test::serial;
const DAY_SECS: i64 = 86400;
@@ -4281,6 +4321,7 @@ mod tests {
/// combination, and must be deterministic: the same input
/// evaluated twice yields an identical event.
#[test]
#[serial]
fn eval_inner_never_panics_and_is_deterministic(
rules in prop::collection::vec(arb_rule(), 0..4),
obj in arb_object_opts(),
@@ -4420,6 +4461,7 @@ mod tests {
/// candidate set — earliest due wins, ties prefer delete-class —
/// and must be `NoneAction` exactly when that set is empty.
#[test]
#[serial]
fn eval_inner_winner_matches_selection_oracle(
rules in prop::collection::vec(arb_selection_rule(), 0..5),
mod_off in 0i64..(2 * DAY_SECS),
@@ -4473,6 +4515,7 @@ mod tests {
/// non-decreasing in `days` (days == 0 maps to UNIX_EPOCH, below
/// any post-1970 deadline).
#[test]
#[serial]
fn expected_expiry_time_is_monotonic_in_days(
mod_off in 0i64..(3650 * DAY_SECS),
d1 in 0i32..2000,
@@ -4494,6 +4537,7 @@ mod tests {
/// to the next whole-day boundary: the result is day-aligned, not
/// before `mod_time + days`, and less than one boundary beyond it.
#[test]
#[serial]
fn expected_expiry_time_lands_on_default_day_boundary(
mod_off in 0i64..(3650 * DAY_SECS),
days in 1i32..2000,
@@ -4511,6 +4555,7 @@ mod tests {
/// to that boundary instead: aligned to it, never early, and less
/// than one boundary late.
#[test]
#[serial]
fn expected_expiry_time_lands_on_explicit_process_boundary(
mod_off in 0i64..(365 * DAY_SECS),
days in 1i32..400,
+6 -30
View File
@@ -12,34 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use std::collections::HashMap;
//! Lifecycle tag decoding reuses the parser owned by `rustfs-replication`.
//!
//! `crates/lifecycle` already depends on `rustfs-replication`, so this is a
//! plain re-export: no new crate edge, and no second copy of the parser to
//! drift from the replication contract.
use url::form_urlencoded;
pub(crate) fn decode_tags_to_map(tags: &str) -> HashMap<String, String> {
let mut list = HashMap::new();
for (k, v) in form_urlencoded::parse(tags.as_bytes()) {
if k.is_empty() {
continue;
}
list.insert(k.to_string(), v.to_string());
}
list
}
#[cfg(test)]
mod tests {
use super::decode_tags_to_map;
#[test]
fn decode_tags_to_map_preserves_bucket_tagging_parser_behavior() {
let tags = decode_tags_to_map("env=prod&encoded=a%2Fb&=ignored");
assert_eq!(tags.get("env").map(String::as_str), Some("prod"));
assert_eq!(tags.get("encoded").map(String::as_str), Some("a/b"));
assert!(!tags.contains_key(""));
}
}
pub(crate) use rustfs_replication::tagging::decode_tags_to_map;
+22 -14
View File
@@ -173,20 +173,28 @@ pub async fn lookup_config() -> Result<Args, OpaConfigError> {
impl AuthZPlugin {
pub fn new(config: Args) -> Self {
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(5))
.connect_timeout(Duration::from_secs(1))
.pool_max_idle_per_host(10)
.pool_idle_timeout(Some(Duration::from_secs(60)))
.tcp_keepalive(Some(Duration::from_secs(30)))
.tcp_nodelay(true)
.http2_keep_alive_interval(Some(Duration::from_secs(30)))
.http2_keep_alive_timeout(Duration::from_secs(15))
.build()
.unwrap_or_else(|err| {
error!("failed to build OPA HTTP client, falling back to default reqwest client: {err}");
reqwest::Client::new()
});
let builder = || {
reqwest::Client::builder()
.timeout(Duration::from_secs(5))
.connect_timeout(Duration::from_secs(1))
.pool_max_idle_per_host(10)
.pool_idle_timeout(Some(Duration::from_secs(60)))
.tcp_keepalive(Some(Duration::from_secs(30)))
.tcp_nodelay(true)
.http2_keep_alive_interval(Some(Duration::from_secs(30)))
.http2_keep_alive_timeout(Duration::from_secs(15))
};
// Never fall back to `reqwest::Client::new()`: it panics for the same
// reason the first build failed (e.g. no system CA bundle, issue
// #6734). Retry with an explicit empty trust store instead — an HTTP
// OPA endpoint keeps working, an HTTPS one fails closed per request.
let client = builder().build().unwrap_or_else(|err| {
error!("failed to build OPA HTTP client ({err}); continuing with an empty trust store");
builder()
.tls_certs_only(std::iter::empty::<reqwest::Certificate>())
.build()
.expect("HTTP client construction must succeed with an explicit empty trust store")
});
Self { client, args: config }
}
+1 -1
View File
@@ -52,4 +52,4 @@ It checks that:
- KMS key ids are derived from each fixture's own `manifest.json`, so local static-KMS runs are not tied to one hard-coded key name
- SSE-C `HEAD` responses round-trip the expected customer algorithm and customer-key MD5
These tests do not yet validate full plaintext reconstruction from MinIO-written encrypted data.
These tests do not validate full plaintext reconstruction from MinIO-written encrypted data — that lives in the reader suite at `rustfs/src/storage/minio_generated_read_test.rs`, run with `--features rio-v2` over the same fixture captures.
+54
View File
@@ -471,4 +471,58 @@ mod tests {
assert_eq!(parsed.delete_markers.len(), 1);
assert_eq!(parsed.delete_markers[0].version_id, "marker-a");
}
// Regression test for backlog#2076: a ListObjectsV2 response for a delimited
// listing over a bucket that holds nested-key objects (e.g. any warm-tier
// target that already stores more than one flat object) includes a
// <CommonPrefixes><Prefix>...</Prefix></CommonPrefixes> element. `CommonPrefix`
// previously had no `rename_all = "PascalCase"`, so quick_xml looked for a
// lowercase `<prefix>` child, never found one, and (with no `#[serde(default)]`
// either) failed the whole response with "missing field `prefix`" — surfacing to
// callers of `WarmBackendS3::in_use()` (tier add/remove) as `TierPermErr`.
#[test]
fn list_objects_v2_xml_parses_common_prefixes() {
let xml = br#"
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>warm-bucket</Name>
<Prefix></Prefix>
<Delimiter>/</Delimiter>
<MaxKeys>1</MaxKeys>
<IsTruncated>false</IsTruncated>
<CommonPrefixes>
<Prefix>subdir/</Prefix>
</CommonPrefixes>
</ListBucketResult>
"#;
let parsed = quick_xml::de::from_reader::<_, ListBucketV2Result>(xml.as_slice()).expect("ListObjectsV2 XML should parse");
assert_eq!(parsed.common_prefixes.len(), 1);
assert_eq!(parsed.common_prefixes[0].prefix, "subdir/");
}
// Same fixture shape as list_object_versions_query hits (ListVersionsResult
// reuses the same CommonPrefix type).
#[test]
fn list_object_versions_xml_parses_common_prefixes() {
let xml = br#"
<ListVersionsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>warm-bucket</Name>
<Prefix></Prefix>
<KeyMarker></KeyMarker>
<VersionIdMarker></VersionIdMarker>
<MaxKeys>1</MaxKeys>
<IsTruncated>false</IsTruncated>
<CommonPrefixes>
<Prefix>subdir/</Prefix>
</CommonPrefixes>
</ListVersionsResult>
"#;
let parsed =
quick_xml::de::from_reader::<_, ListVersionsResult>(xml.as_slice()).expect("ListObjectVersions XML should parse");
assert_eq!(parsed.common_prefixes.len(), 1);
assert_eq!(parsed.common_prefixes[0].prefix, "subdir/");
}
}
+1 -1
View File
@@ -24,6 +24,7 @@ use std::{collections::HashMap, sync::Arc};
use time::{Duration, OffsetDateTime, macros::format_description};
use tracing::{error, info, warn};
use rustfs_utils::http::{is_amz_header, is_minio_header, is_rustfs_header, is_standard_header, is_storageclass_header};
use s3s::dto::{ObjectLockLegalHoldStatus, ObjectLockRetentionMode, ReplicationStatus};
use s3s::header::{
X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, X_AMZ_REPLICATION_STATUS,
@@ -40,7 +41,6 @@ use crate::{
constants::{ISO8601_DATEFORMAT, MAX_MULTIPART_PUT_OBJECT_SIZE, MIN_PART_SIZE},
credentials::SignatureType,
transition_api::{ReaderImpl, TransitionClient, UploadInfo},
utils::{is_amz_header, is_minio_header, is_rustfs_header, is_standard_header, is_storageclass_header},
};
#[derive(Debug, Clone)]
+28
View File
@@ -30,6 +30,7 @@ use crate::utils::base64_decode;
use super::transition_api;
#[derive(Debug, Default, Serialize, Deserialize)]
#[serde(default, rename_all = "PascalCase")]
pub struct CommonPrefix {
pub prefix: String,
}
@@ -395,3 +396,30 @@ impl DeleteMultiObjects {
})
}
}
#[cfg(test)]
mod tests {
use super::ListBucketV2Result;
#[test]
fn list_bucket_v2_common_prefix_accepts_s3_pascal_case_xml() {
let xml = r#"
<ListBucketResult>
<Name>tier-bucket</Name>
<Prefix></Prefix>
<KeyCount>1</KeyCount>
<MaxKeys>1000</MaxKeys>
<Delimiter>/</Delimiter>
<IsTruncated>false</IsTruncated>
<CommonPrefixes>
<Prefix>tenant-a/</Prefix>
</CommonPrefixes>
</ListBucketResult>
"#;
let result = quick_xml::de::from_str::<ListBucketV2Result>(xml).expect("S3 list response should decode");
assert_eq!(result.common_prefixes.len(), 1);
assert_eq!(result.common_prefixes[0].prefix, "tenant-a/");
}
}
-61
View File
@@ -12,67 +12,6 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use lazy_static::lazy_static;
use std::collections::HashMap;
use s3s::header::X_AMZ_STORAGE_CLASS;
lazy_static! {
static ref SUPPORTED_QUERY_VALUES: HashMap<String, bool> = {
let mut m = HashMap::new();
m.insert("attributes".to_string(), true);
m.insert("partNumber".to_string(), true);
m.insert("versionId".to_string(), true);
m.insert("response-cache-control".to_string(), true);
m.insert("response-content-disposition".to_string(), true);
m.insert("response-content-encoding".to_string(), true);
m.insert("response-content-language".to_string(), true);
m.insert("response-content-type".to_string(), true);
m.insert("response-expires".to_string(), true);
m
};
static ref SUPPORTED_HEADERS: HashMap<String, bool> = {
let mut m = HashMap::new();
m.insert("content-type".to_string(), true);
m.insert("cache-control".to_string(), true);
m.insert("content-encoding".to_string(), true);
m.insert("content-disposition".to_string(), true);
m.insert("content-language".to_string(), true);
m.insert("x-amz-website-redirect-location".to_string(), true);
m.insert("x-amz-object-lock-mode".to_string(), true);
m.insert("x-amz-metadata-directive".to_string(), true);
m.insert("x-amz-object-lock-retain-until-date".to_string(), true);
m.insert("expires".to_string(), true);
m.insert("x-amz-replication-status".to_string(), true);
m
};
}
pub fn is_storageclass_header(header_key: &str) -> bool {
header_key.to_lowercase() == X_AMZ_STORAGE_CLASS.as_str().to_lowercase()
}
pub fn is_standard_header(header_key: &str) -> bool {
*SUPPORTED_HEADERS.get(&header_key.to_lowercase()).unwrap_or(&false)
}
pub fn is_amz_header(header_key: &str) -> bool {
let key = header_key.to_lowercase();
key.starts_with("x-amz-meta-")
|| key.starts_with("x-amz-grant-")
|| key == "x-amz-acl"
|| rustfs_utils::http::is_sse_header(header_key)
|| key.starts_with("x-amz-checksum-")
}
pub fn is_rustfs_header(header_key: &str) -> bool {
header_key.to_lowercase().starts_with("x-rustfs-")
}
pub fn is_minio_header(header_key: &str) -> bool {
header_key.to_lowercase().starts_with("x-minio-")
}
/// Standard base64 (with `+`/`/` and `=` padding). Every base64 value this
/// transition client emits or parses — `Content-MD5`, `x-amz-checksum-*`, and
/// checksum digests in request/response bodies — is S3 wire format, which is
+2 -1
View File
@@ -105,7 +105,8 @@ hex-simd.workspace = true
[dev-dependencies]
tracing-subscriber = { workspace = true, features = ["json", "env-filter", "time"] }
temp-env = { workspace = true, features = ["async_closure"] }
serial_test = { workspace = true }
temp-env = { workspace = true }
tempfile = { workspace = true }
uuid = { workspace = true, features = ["v4", "serde", "fast-rng", "macro-diagnostics"] }
tokio = { workspace = true, features = ["test-util", "fs", "rt-multi-thread"] }
+5
View File
@@ -951,8 +951,10 @@ impl ScannerConfigObjectDelete for SetDisks {
#[cfg(test)]
mod tests {
use super::*;
use serial_test::serial;
#[tokio::test]
#[serial]
async fn runtime_tier_names_serves_cached_arc_within_ttl() {
reset_tier_name_cache_for_test();
// The tier config manager is unconfigured in unit tests, so the
@@ -1028,6 +1030,7 @@ mod tests {
}
#[test]
#[serial]
fn foreground_read_guard_tracks_stream_lifetime() {
reset_foreground_read_activity_for_test();
assert_eq!(current_foreground_read_activity(), 0);
@@ -1041,6 +1044,7 @@ mod tests {
}
#[test]
#[serial]
fn foreground_read_activity_keeps_larger_signal() {
reset_foreground_read_activity_for_test();
let _guard = ForegroundReadGuard::new();
@@ -1053,6 +1057,7 @@ mod tests {
}
#[test]
#[serial]
fn scanner_runtime_guard_tracks_runtime_lifetime() {
reset_scanner_runtime_instances_for_test();
assert!(!scanner_runtime_initialized());
+14
View File
@@ -896,6 +896,7 @@ mod tests {
SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION,
SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE, SCANNER_SPEED, SCANNER_SUB_SYS, ScannerSpeed,
};
use serial_test::serial;
use std::collections::HashMap;
use std::time::Duration;
use temp_env::{with_var, with_var_unset};
@@ -943,6 +944,7 @@ mod tests {
}
#[test]
#[serial]
fn scanner_runtime_config_uses_persisted_values_when_env_is_unset() {
let config = server_config_with_scanner(&[
(SCANNER_SPEED, "slow"),
@@ -1014,6 +1016,7 @@ mod tests {
}
#[test]
#[serial]
fn scanner_runtime_config_normalizes_persisted_default_speed() {
let config = server_config_with_scanner(&[(SCANNER_SPEED, "default")]);
@@ -1029,6 +1032,7 @@ mod tests {
}
#[test]
#[serial]
fn scanner_runtime_config_prefers_env_over_persisted_config() {
let config = server_config_with_scanner(&[(SCANNER_SPEED, "slowest"), (SCANNER_CYCLE, "600")]);
@@ -1045,6 +1049,7 @@ mod tests {
}
#[test]
#[serial]
fn scanner_runtime_config_prefers_heal_bitrot_cycle_over_scanner_compat_config() {
let config = server_config_with_scanner_and_heal(&[(SCANNER_BITROT_CYCLE, "3600")], &[(HEAL_BITROT_CYCLE, "off")]);
@@ -1057,6 +1062,7 @@ mod tests {
}
#[test]
#[serial]
fn scanner_runtime_config_marks_scanner_bitrot_cycle_as_compat_source() {
let config = server_config_with_scanner(&[(SCANNER_BITROT_CYCLE, "3600")]);
@@ -1073,6 +1079,7 @@ mod tests {
}
#[test]
#[serial]
fn scanner_runtime_config_normalizes_persisted_default_bitrot_cycles() {
let default_cycle = DEFAULT_HEAL_BITROT_CYCLE_SECS.to_string();
for config in [
@@ -1097,6 +1104,7 @@ mod tests {
}
#[test]
#[serial]
fn scanner_runtime_config_validation_rejects_invalid_persisted_speed_with_env_override() {
let config = server_config_with_scanner(&[(SCANNER_SPEED, "warp")]);
@@ -1130,6 +1138,7 @@ mod tests {
}
#[test]
#[serial]
fn scanner_runtime_config_uses_derived_delay_for_excessive_env_override() {
let config = server_config_with_scanner(&[(SCANNER_SPEED, "slow")]);
@@ -1150,6 +1159,7 @@ mod tests {
}
#[test]
#[serial]
fn scanner_runtime_config_status_reports_value_sources() {
let config = server_config_with_scanner(&[(SCANNER_CYCLE_MAX_OBJECTS, "100"), (SCANNER_CACHE_SAVE_TIMEOUT, "5")]);
@@ -1170,6 +1180,7 @@ mod tests {
}
#[test]
#[serial]
fn applied_runtime_config_is_the_authoritative_scheduler_state() {
let config = server_config_with_scanner(&[(SCANNER_CYCLE, "321")]);
@@ -1186,6 +1197,7 @@ mod tests {
}
#[test]
#[serial]
fn scanner_runtime_config_status_reports_persisted_pacing_overrides() {
let config = server_config_with_scanner(&[("delay", "3.5"), ("max_wait", "7")]);
@@ -1207,6 +1219,7 @@ mod tests {
}
#[test]
#[serial]
fn scanner_runtime_config_status_prefers_env_pacing_overrides() {
let config = server_config_with_scanner(&[("delay", "3.5"), ("max_wait", "7")]);
@@ -1228,6 +1241,7 @@ mod tests {
}
#[test]
#[serial]
fn scanner_runtime_config_status_preserves_subsecond_max_wait() {
let config = server_config_with_scanner(&[(SCANNER_SPEED, "fast")]);
+56
View File
@@ -21,6 +21,7 @@ use crate::{
ScannerPutObjReader as PutObjReader, init_bucket_metadata_sys_for_scanner_tests, init_ecstore_config_for_scanner_tests,
init_local_disks_with_instance_ctx,
};
use serial_test::serial;
use std::collections::{HashMap, HashSet};
use std::io::Cursor;
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
@@ -587,6 +588,7 @@ fn test_initial_scanner_delay_uses_configured_start_delay() {
}
#[test]
#[serial]
fn test_initial_scanner_delay_uses_cycle_without_explicit_start_delay() {
with_var(ENV_SCANNER_CYCLE, Some("120"), || {
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
@@ -633,6 +635,7 @@ fn test_initial_scanner_delay_keeps_delay_for_replication_without_buckets() {
}
#[test]
#[serial]
fn test_scanner_cycle_max_duration_uses_env() {
with_var(ENV_SCANNER_CYCLE_MAX_DURATION_SECS, Some("42"), || {
assert_eq!(scanner_cycle_max_duration(), Some(Duration::from_secs(42)));
@@ -676,6 +679,7 @@ async fn test_scanner_cycle_budget_drop_cancels_child_without_elapsed() {
}
#[test]
#[serial]
fn test_scanner_cycle_budget_config_uses_work_budget_env() {
with_var(ENV_SCANNER_CYCLE_MAX_OBJECTS, Some("100"), || {
with_var(ENV_SCANNER_CYCLE_MAX_DIRECTORIES, Some("25"), || {
@@ -687,6 +691,7 @@ fn test_scanner_cycle_budget_config_uses_work_budget_env() {
}
#[test]
#[serial]
fn test_scanner_cycle_budget_config_disables_zero_work_budgets() {
with_var(ENV_SCANNER_CYCLE_MAX_OBJECTS, Some("0"), || {
with_var(ENV_SCANNER_CYCLE_MAX_DIRECTORIES, Some("0"), || {
@@ -729,6 +734,7 @@ fn test_scan_cycle_partial_source_maps_budget_reason() {
}
#[tokio::test]
#[serial]
async fn test_mark_scan_cycle_idle_clears_published_cycle_state() {
let mut cycle_info = CurrentCycle {
current: 12,
@@ -757,6 +763,7 @@ async fn test_mark_scan_cycle_idle_clears_published_cycle_state() {
}
#[tokio::test]
#[serial]
async fn scanner_cycle_metrics_guard_covers_published_first_cycle_lifetime() {
let cycle_started = Utc::now() - chrono::Duration::seconds(5);
let mut cycle_info = CurrentCycle {
@@ -783,6 +790,7 @@ async fn scanner_cycle_metrics_guard_covers_published_first_cycle_lifetime() {
}
#[tokio::test]
#[serial]
async fn scanner_cycle_metrics_guard_keeps_active_cycle_published_during_finalization() {
let mut cycle_info = CurrentCycle {
current: 12,
@@ -807,6 +815,7 @@ async fn scanner_cycle_metrics_guard_keeps_active_cycle_published_during_finaliz
}
#[tokio::test]
#[serial]
async fn scanner_cycle_metrics_guard_drop_clears_activity() {
let guard = ScannerCycleMetricsGuard::new(CurrentCycle {
current: 12,
@@ -824,6 +833,7 @@ async fn scanner_cycle_metrics_guard_drop_clears_activity() {
}
#[tokio::test]
#[serial]
async fn run_data_scanner_cycle_publishes_activity_for_owner_lifetime() {
let (_temp_dir, store) = setup_scanner_cycle_store().await;
let ctx = CancellationToken::new();
@@ -874,6 +884,7 @@ async fn run_data_scanner_cycle_publishes_activity_for_owner_lifetime() {
}
#[tokio::test]
#[serial]
async fn test_finalize_partial_scan_cycle_advances_and_persists_counter() {
let store = Arc::new(MemoryConfigStore::default());
let ctx = CancellationToken::new();
@@ -909,6 +920,7 @@ async fn test_finalize_partial_scan_cycle_advances_and_persists_counter() {
}
#[tokio::test]
#[serial]
async fn scanner_cycle_recovers_to_newer_durable_cache_floor() {
let store = Arc::new(MemoryConfigStore::default());
let ctx = CancellationToken::new();
@@ -948,6 +960,7 @@ async fn scanner_cycle_recovers_to_newer_durable_cache_floor() {
}
#[tokio::test]
#[serial]
async fn scanner_cycle_rejects_invalid_cache_floor() {
let store = Arc::new(MemoryConfigStore::default());
let ctx = CancellationToken::new();
@@ -2369,6 +2382,7 @@ async fn scanner_usage_bootstrap_allows_first_bucket_to_win_startup() {
}
#[tokio::test]
#[serial]
async fn scanner_usage_backup_uses_durable_cycle_cadence_across_tasks() {
let store = Arc::new(MemoryConfigStore::default());
let ctx = CancellationToken::new();
@@ -2489,6 +2503,7 @@ fn scanner_cycle_advance_fails_before_reserved_exhausted_value() {
}
#[tokio::test]
#[serial]
async fn test_finalize_partial_scan_cycle_reports_persist_failure() {
let store = Arc::new(MemoryConfigStore::default());
let ctx = CancellationToken::new();
@@ -2512,6 +2527,7 @@ async fn test_finalize_partial_scan_cycle_reports_persist_failure() {
}
#[tokio::test]
#[serial]
async fn test_persist_scanner_cycle_state_reconciles_newer_winner() {
let store = Arc::new(MemoryConfigStore::default());
let ctx = CancellationToken::new();
@@ -3167,6 +3183,7 @@ async fn test_observational_usage_defers_when_authoritative_baseline_is_missing(
}
#[tokio::test]
#[serial]
async fn test_usage_route_barrier_precedes_durable_reconciliation() {
let store = Arc::new(MemoryConfigStore::default());
let key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_OBJ_NAME_PATH.as_str());
@@ -3235,6 +3252,7 @@ async fn coordinator_does_not_put_after_remote_generation_flip() {
}
#[tokio::test]
#[serial]
async fn test_deferred_usage_save_keeps_last_real_save_metric() {
let metrics = global_metrics();
metrics.record_scanner_usage_save_result(ScannerUsageSaveResult::Success);
@@ -4307,6 +4325,7 @@ fn scanner_cycle_cache_floor_stays_pending_during_deferred_usage_publication() {
}
#[test]
#[serial]
fn finalizing_a_saved_cycle_acknowledges_its_exact_dirty_snapshot() {
crate::scanner_io::clear_dirty_usage_bucket("photos");
crate::scanner_io::record_dirty_usage_bucket("photos");
@@ -4333,6 +4352,7 @@ fn finalizing_a_saved_cycle_acknowledges_its_exact_dirty_snapshot() {
}
#[test]
#[serial]
fn finalizing_a_deferred_usage_save_keeps_dirty_work_pending() {
crate::scanner_io::clear_dirty_usage_bucket("photos");
crate::scanner_io::record_dirty_usage_bucket("photos");
@@ -4375,6 +4395,7 @@ async fn scanner_cycle_keeps_remote_pending_acknowledgement() {
}
#[test]
#[serial]
fn finalizing_an_already_durable_cycle_acknowledges_its_exact_dirty_snapshot() {
crate::scanner_io::clear_dirty_usage_bucket("photos");
crate::scanner_io::record_dirty_usage_bucket("photos");
@@ -4389,6 +4410,7 @@ fn finalizing_an_already_durable_cycle_acknowledges_its_exact_dirty_snapshot() {
}
#[test]
#[serial]
fn finalizing_a_prior_same_cycle_snapshot_keeps_new_dirty_work_pending() {
crate::scanner_io::clear_dirty_usage_bucket("photos");
crate::scanner_io::record_dirty_usage_bucket("photos");
@@ -4404,6 +4426,7 @@ fn finalizing_a_prior_same_cycle_snapshot_keeps_new_dirty_work_pending() {
}
#[test]
#[serial]
fn finalizing_a_durable_superseded_snapshot_keeps_dirty_work_pending() {
crate::scanner_io::clear_dirty_usage_bucket("photos");
crate::scanner_io::record_dirty_usage_bucket("photos");
@@ -4419,6 +4442,7 @@ fn finalizing_a_durable_superseded_snapshot_keeps_dirty_work_pending() {
}
#[test]
#[serial]
fn data_usage_persist_wait_covers_cache_retries_and_backup() {
with_var(rustfs_config::ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, Some("7"), || {
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
@@ -4471,6 +4495,7 @@ async fn maintenance_feature_inspection_preserves_base_cycle_after_timeout() {
}
#[tokio::test(start_paused = true)]
#[serial]
async fn stable_maintenance_detection_preserves_base_cycle_after_timeout() {
let ctx = CancellationToken::new();
@@ -4536,6 +4561,7 @@ async fn maintenance_feature_inspection_stops_on_cancellation() {
}
#[test]
#[serial]
fn test_cycle_interval_prefers_explicit_cycle_override() {
with_var(ENV_SCANNER_SPEED, Some("slowest"), || {
with_var(ENV_SCANNER_CYCLE, Some("42"), || {
@@ -4545,6 +4571,7 @@ fn test_cycle_interval_prefers_explicit_cycle_override() {
}
#[test]
#[serial]
fn test_cycle_interval_prefers_explicit_cycle_over_default_cycle() {
let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS);
@@ -4554,6 +4581,7 @@ fn test_cycle_interval_prefers_explicit_cycle_over_default_cycle() {
}
#[test]
#[serial]
fn test_cycle_interval_uses_scanner_default_speed_override_when_unconfigured() {
let _guard = ScannerDefaultSpeedGuard::set(ScannerSpeed::Slowest);
@@ -4563,6 +4591,7 @@ fn test_cycle_interval_uses_scanner_default_speed_override_when_unconfigured() {
}
#[test]
#[serial]
fn test_cycle_interval_prefers_explicit_speed_over_default_speed_override() {
let _guard = ScannerDefaultSpeedGuard::set(ScannerSpeed::Slowest);
@@ -4580,6 +4609,7 @@ fn test_cycle_interval_prefers_explicit_speed_over_default_speed_override() {
}
#[test]
#[serial]
fn test_cycle_interval_uses_default_cycle_override_when_unconfigured() {
let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS);
@@ -4761,6 +4791,7 @@ fn scanner_cycle_wait_plan_drives_growth_resets_and_bitrot_cap() {
}
#[test]
#[serial]
fn scanner_cycle_schedule_status_reports_effective_backoff() {
record_scanner_cycle_schedule(Duration::from_millis(86_400_001), true, 2_048, true, 7);
@@ -5077,6 +5108,7 @@ fn dirty_usage_wakes_are_disabled_for_explicit_cycle_policy() {
}
#[test]
#[serial]
fn clean_idle_cap_preserves_default_bitrot_coverage_window() {
let config = ScannerRuntimeConfig {
bitrot_cycle: Some(Duration::from_secs(30 * 24 * 60 * 60)),
@@ -5106,6 +5138,7 @@ fn clean_idle_cap_allows_policy_max_when_bitrot_is_disabled() {
}
#[test]
#[serial]
fn clean_idle_cap_never_shortens_the_base_cycle() {
let config = ScannerRuntimeConfig {
bitrot_cycle: Some(Duration::from_secs(60)),
@@ -5119,6 +5152,7 @@ fn clean_idle_cap_never_shortens_the_base_cycle() {
}
#[test]
#[serial]
fn test_cycle_interval_keeps_default_cycle_with_explicit_speed() {
let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS);
@@ -5136,6 +5170,7 @@ fn test_cycle_interval_keeps_default_cycle_with_explicit_speed() {
}
#[test]
#[serial]
fn test_cycle_interval_prefers_explicit_start_delay_over_default_cycle() {
let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS);
@@ -5149,6 +5184,7 @@ fn test_cycle_interval_prefers_explicit_start_delay_over_default_cycle() {
}
#[test]
#[serial]
fn test_cycle_interval_supports_minio_speed_alias() {
with_var_unset(ENV_SCANNER_SPEED, || {
with_var_unset(ENV_SCANNER_CYCLE, || {
@@ -5162,6 +5198,7 @@ fn test_cycle_interval_supports_minio_speed_alias() {
}
#[test]
#[serial]
fn test_cycle_interval_supports_minio_cycle_alias() {
with_var_unset(ENV_SCANNER_CYCLE, || {
with_var_unset(ENV_SCANNER_START_DELAY_SECS, || {
@@ -5181,6 +5218,7 @@ fn test_randomized_cycle_delay_handles_small_start_delay() {
}
#[tokio::test]
#[serial]
async fn test_wait_for_next_scanner_cycle_wakes_for_dirty_usage() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
@@ -5206,6 +5244,7 @@ async fn test_wait_for_next_scanner_cycle_wakes_for_dirty_usage() {
}
#[tokio::test]
#[serial]
async fn test_wait_for_next_scanner_cycle_sees_unattempted_dirty_usage() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let dirty_generation = crate::scanner_io::dirty_usage_generation();
@@ -5227,6 +5266,7 @@ async fn test_wait_for_next_scanner_cycle_sees_unattempted_dirty_usage() {
}
#[tokio::test(start_paused = true)]
#[serial]
async fn test_wait_for_next_scanner_cycle_retries_stable_dirty_usage_on_timer() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
crate::scanner_io::record_dirty_usage_bucket("photos");
@@ -5248,6 +5288,7 @@ async fn test_wait_for_next_scanner_cycle_retries_stable_dirty_usage_on_timer()
}
#[tokio::test(start_paused = true)]
#[serial]
async fn test_wait_for_next_scanner_cycle_can_defer_dirty_wakes_until_timer() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let ctx = CancellationToken::new();
@@ -5266,6 +5307,7 @@ async fn test_wait_for_next_scanner_cycle_can_defer_dirty_wakes_until_timer() {
}
#[tokio::test]
#[serial]
async fn test_wait_for_next_scanner_cycle_wakes_for_repeated_dirty_bucket() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
crate::scanner_io::record_dirty_usage_bucket("photos");
@@ -5291,6 +5333,7 @@ async fn test_wait_for_next_scanner_cycle_wakes_for_repeated_dirty_bucket() {
}
#[tokio::test]
#[serial]
async fn test_wait_for_next_scanner_cycle_reschedules_for_runtime_config() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let observed_generation = crate::runtime_config::scanner_runtime_config_generation();
@@ -5318,6 +5361,7 @@ async fn test_wait_for_next_scanner_cycle_reschedules_for_runtime_config() {
}
#[tokio::test]
#[serial]
async fn test_wait_for_next_scanner_cycle_reschedules_for_maintenance_change() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let observed_generation = crate::scanner_io::scanner_maintenance_generation();
@@ -5629,6 +5673,7 @@ fn scanner_activity_after_a_cycle_restores_the_base_interval() {
}
#[tokio::test(start_paused = true)]
#[serial]
async fn distributed_clean_idle_wait_wakes_at_base_interval_for_remote_activity() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let ctx = CancellationToken::new();
@@ -5656,6 +5701,7 @@ async fn distributed_clean_idle_wait_wakes_at_base_interval_for_remote_activity(
}
#[tokio::test(start_paused = true)]
#[serial]
async fn superseded_retry_wait_defers_dirty_cluster_activity_until_timer() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let ctx = CancellationToken::new();
@@ -5751,6 +5797,7 @@ async fn superseded_retry_wait_wakes_when_remote_restart_clears_movement_state()
}
#[tokio::test(start_paused = true)]
#[serial]
async fn distributed_clean_idle_wait_blocks_backoff_for_unpropagated_maintenance() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let ctx = CancellationToken::new();
@@ -5777,6 +5824,7 @@ async fn distributed_clean_idle_wait_blocks_backoff_for_unpropagated_maintenance
}
#[tokio::test(start_paused = true)]
#[serial]
async fn distributed_clean_idle_wait_fails_closed_when_a_peer_is_unverifiable() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let ctx = CancellationToken::new();
@@ -5803,6 +5851,7 @@ async fn distributed_clean_idle_wait_fails_closed_when_a_peer_is_unverifiable()
}
#[tokio::test(start_paused = true)]
#[serial]
async fn distributed_clean_idle_wait_keeps_the_extended_deadline_when_peers_are_clean() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let ctx = CancellationToken::new();
@@ -5830,6 +5879,7 @@ async fn distributed_clean_idle_wait_keeps_the_extended_deadline_when_peers_are_
}
#[tokio::test(start_paused = true)]
#[serial]
async fn scanner_activity_probe_wait_is_cancellation_aware() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let ctx = CancellationToken::new();
@@ -5860,6 +5910,7 @@ async fn scanner_activity_probe_wait_is_cancellation_aware() {
}
#[tokio::test(start_paused = true)]
#[serial]
async fn scanner_activity_probe_wait_stops_after_leader_lock_loss() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let ctx = CancellationToken::new();
@@ -5891,6 +5942,7 @@ async fn scanner_activity_probe_wait_stops_after_leader_lock_loss() {
}
#[test]
#[serial]
fn test_get_cycle_scan_mode_runs_deep_until_selection_window_completes() {
with_var(ENV_SCANNER_BITROT_CYCLE_SECS, Some("3600"), || {
let mode = get_cycle_scan_mode(10, 0, Some(Utc::now()), bitrot_scan_cycle());
@@ -5899,6 +5951,7 @@ fn test_get_cycle_scan_mode_runs_deep_until_selection_window_completes() {
}
#[test]
#[serial]
fn test_get_cycle_scan_mode_respects_elapsed_bitrot_cycle() {
with_var(ENV_SCANNER_BITROT_CYCLE_SECS, Some("3600"), || {
let recent = Utc::now() - chrono::Duration::minutes(30);
@@ -5910,6 +5963,7 @@ fn test_get_cycle_scan_mode_respects_elapsed_bitrot_cycle() {
}
#[test]
#[serial]
fn test_get_cycle_scan_mode_can_disable_periodic_deep_scan() {
with_var(ENV_SCANNER_BITROT_CYCLE_SECS, Some("off"), || {
assert_eq!(get_cycle_scan_mode(1, 0, None, bitrot_scan_cycle()), HealScanMode::Normal);
@@ -5917,6 +5971,7 @@ fn test_get_cycle_scan_mode_can_disable_periodic_deep_scan() {
}
#[test]
#[serial]
fn test_background_heal_info_for_scan_start_marks_deep_active() {
let now = Utc::now();
let info =
@@ -5942,6 +5997,7 @@ fn background_heal_read_failures_never_become_initializable_defaults() {
}
#[test]
#[serial]
fn test_background_heal_info_for_scan_start_keeps_deep_window_start() {
with_var_unset(ENV_SCANNER_BITROT_CYCLE_SECS, || {
let started_at = Utc::now();
@@ -18,6 +18,7 @@ use super::*;
use crate::storage_api::VersionPurgeStatusType;
use crate::{DiskOption, Endpoint, STORAGE_FORMAT_FILE, TierStats, new_disk, storageclass};
use rustfs_filemeta::{FileInfo, FileMeta, MetadataResolutionParams};
use serial_test::serial;
use std::io::Write;
#[cfg(unix)]
use std::os::unix::fs::{PermissionsExt, symlink};
@@ -375,6 +376,7 @@ impl Drop for TestGuard {
}
#[tokio::test]
#[serial]
async fn test_should_skip_failed_respects_ttl() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir);
@@ -396,6 +398,7 @@ async fn test_should_skip_failed_respects_ttl() {
}
#[tokio::test]
#[serial]
async fn test_record_failed_ttl_zero_noop() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(0, 100, &mut scanner, temp_dir);
@@ -549,6 +552,7 @@ fn test_should_account_replication_stats_only_for_live_object_versions() {
}
#[tokio::test]
#[serial]
async fn test_heal_replication_only_queues_pending_null_deletes() {
async fn replication_skipped_count() -> u64 {
global_metrics()
@@ -797,6 +801,7 @@ async fn test_scanner_heal_admission_accounting_maps_deep_scan_to_bitrot() {
}
#[test]
#[serial]
fn test_excessive_version_alert_thresholds_use_env() {
with_var(rustfs_config::ENV_SCANNER_ALERT_EXCESS_VERSIONS, Some("3"), || {
with_var(rustfs_config::ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE, Some("100"), || {
@@ -811,6 +816,7 @@ fn test_excessive_version_alert_thresholds_use_env() {
}
#[test]
#[serial]
fn test_excessive_folders_threshold_uses_env() {
with_var(rustfs_config::ENV_SCANNER_ALERT_EXCESS_FOLDERS, Some("3"), || {
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
@@ -820,6 +826,7 @@ fn test_excessive_folders_threshold_uses_env() {
}
#[test]
#[serial]
fn test_excessive_folders_threshold_default_supports_pbs_layout() {
with_var_unset(rustfs_config::ENV_SCANNER_ALERT_EXCESS_FOLDERS, || {
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
@@ -829,6 +836,7 @@ fn test_excessive_folders_threshold_default_supports_pbs_layout() {
}
#[test]
#[serial]
fn test_scanner_yield_every_n_objects_uses_env() {
with_var(rustfs_config::ENV_SCANNER_YIELD_EVERY_N_OBJECTS, Some("32"), || {
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
@@ -838,6 +846,7 @@ fn test_scanner_yield_every_n_objects_uses_env() {
}
#[test]
#[serial]
fn test_scanner_yield_every_n_objects_uses_default() {
with_var_unset(rustfs_config::ENV_SCANNER_YIELD_EVERY_N_OBJECTS, || {
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
@@ -964,6 +973,7 @@ fn test_order_folders_for_resume_reports_stale_hint() {
}
#[tokio::test]
#[serial]
async fn test_record_failed_prunes_to_max_entries() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(1000, 2, &mut scanner, temp_dir);
@@ -995,6 +1005,7 @@ async fn test_record_failed_prunes_to_max_entries() {
}
#[tokio::test]
#[serial]
async fn test_prune_failed_objects_cache_drops_expired() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(5, 10, &mut scanner, temp_dir);
@@ -1018,6 +1029,7 @@ async fn test_prune_failed_objects_cache_drops_expired() {
}
#[tokio::test]
#[serial]
async fn test_prune_failed_objects_max_zero_keeps_fresh() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 0, &mut scanner, temp_dir);
@@ -1840,6 +1852,7 @@ async fn test_heal_actions_returns_actual_size_without_inline_heal() {
#[tokio::test]
#[cfg(unix)]
#[serial]
async fn test_scan_folder_skips_unreadable_child_directory() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 0, &mut scanner, temp_dir.clone());
@@ -1871,6 +1884,7 @@ async fn test_scan_folder_skips_unreadable_child_directory() {
}
#[tokio::test]
#[serial]
async fn test_scan_folder_exits_when_abandoned_child_listing_finishes() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -1984,6 +1998,7 @@ async fn test_scan_folder_exits_when_abandoned_child_listing_finishes() {
}
#[tokio::test]
#[serial]
async fn test_scan_folder_xl_meta_named_directory_uses_namespace_descent() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -2029,6 +2044,7 @@ async fn test_scan_folder_xl_meta_named_directory_uses_namespace_descent() {
}
#[tokio::test(flavor = "current_thread")]
#[serial]
async fn test_scan_folder_corrupt_xl_meta_stops_erasure_data_dir_descent() {
let logs = CapturedLogs::default();
let subscriber = tracing_subscriber::fmt()
@@ -2190,6 +2206,7 @@ async fn test_scan_folder_corrupt_xl_meta_stops_erasure_data_dir_descent() {
}
#[tokio::test]
#[serial]
async fn test_scan_folder_missing_xl_meta_stops_erasure_data_dir_descent() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -2267,6 +2284,7 @@ async fn test_scan_folder_missing_xl_meta_stops_erasure_data_dir_descent() {
}
#[tokio::test]
#[serial]
async fn test_scan_folder_uuid_namespace_part_name_directory_is_not_data_dir() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -2328,6 +2346,7 @@ async fn test_scan_folder_uuid_namespace_part_name_directory_is_not_data_dir() {
}
#[tokio::test]
#[serial]
async fn test_scan_folder_non_erasure_metadata_keeps_namespace_descent() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -2369,6 +2388,7 @@ async fn test_scan_folder_non_erasure_metadata_keeps_namespace_descent() {
}
#[tokio::test]
#[serial]
async fn test_scan_folder_compacted_parent_sends_partial_update() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -2410,6 +2430,7 @@ async fn test_scan_folder_compacted_parent_sends_partial_update() {
}
#[tokio::test]
#[serial]
async fn test_scan_data_folder_cancelled_before_scan_clears_current_path() {
let (scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard {
@@ -2454,6 +2475,7 @@ async fn test_scan_data_folder_cancelled_before_scan_clears_current_path() {
}
#[tokio::test]
#[serial]
async fn test_scan_data_folder_returns_partial_cache_on_budget_cancel() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -2509,6 +2531,7 @@ async fn test_scan_data_folder_returns_partial_cache_on_budget_cancel() {
}
#[tokio::test]
#[serial]
async fn test_scan_data_folder_reports_invalid_checkpoint_ignored_once() {
let (scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard {
@@ -2553,6 +2576,7 @@ async fn test_scan_data_folder_reports_invalid_checkpoint_ignored_once() {
}
#[tokio::test]
#[serial]
async fn test_scan_data_folder_resume_hint_prioritizes_next_existing_folder() {
let (scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard {
@@ -2626,6 +2650,7 @@ async fn test_scan_data_folder_resume_hint_prioritizes_next_existing_folder() {
}
#[tokio::test]
#[serial]
async fn scan_data_folder_missing_bucket_returns_partial() {
let (scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard {
@@ -2677,6 +2702,7 @@ async fn scan_data_folder_missing_bucket_returns_partial() {
}
#[tokio::test]
#[serial]
async fn scan_data_folder_missing_scan_root_returns_partial() {
let (scanner, temp_dir) = build_test_scanner().await;
tokio::fs::remove_dir_all(&temp_dir)
@@ -2722,6 +2748,7 @@ async fn scan_data_folder_missing_scan_root_returns_partial() {
}
#[tokio::test]
#[serial]
async fn test_scan_data_folder_resume_hint_orders_across_new_and_existing_folders() {
let (scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard {
@@ -2790,6 +2817,7 @@ async fn test_scan_data_folder_resume_hint_orders_across_new_and_existing_folder
}
#[tokio::test]
#[serial]
async fn test_scan_data_folder_partial_object_budget_accumulates_progress() {
let (scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard {
@@ -2872,6 +2900,7 @@ async fn test_scan_data_folder_partial_object_budget_accumulates_progress() {
}
#[tokio::test]
#[serial]
async fn test_partial_compacted_entry_does_not_carry_children() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard {
@@ -2917,6 +2946,7 @@ async fn test_partial_compacted_entry_does_not_carry_children() {
}
#[tokio::test]
#[serial]
async fn test_partial_entry_does_not_carry_missing_old_child() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard {
@@ -2949,6 +2979,7 @@ async fn test_partial_entry_does_not_carry_missing_old_child() {
}
#[tokio::test]
#[serial]
async fn test_legacy_windows_cache_rebuilds_and_round_trips_portable_keys() {
let (scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard {
@@ -3015,6 +3046,7 @@ async fn test_legacy_windows_cache_rebuilds_and_round_trips_portable_keys() {
}
#[tokio::test]
#[serial]
async fn test_scan_data_folder_success_clears_resume_hint() {
let (scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard {
@@ -3057,6 +3089,7 @@ async fn test_scan_data_folder_success_clears_resume_hint() {
}
#[tokio::test]
#[serial]
async fn test_scan_data_folder_keeps_unresolved_objects_partial() {
let (scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard {
@@ -3104,6 +3137,7 @@ async fn test_scan_data_folder_keeps_unresolved_objects_partial() {
#[tokio::test]
#[cfg(unix)]
#[serial]
async fn test_scan_folder_ignores_symlinked_child_directory() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 0, &mut scanner, temp_dir.clone());

Some files were not shown because too many files have changed in this diff Show More