Commit Graph
100 Commits
Author SHA1 Message Date
Zhengchao AnandGitHub 1585308f0f fix(test): restore #[serial] markers the fallback runner still needs (#6767) 2026-08-28 08:41:36 +08:00
Zhengchao AnandGitHub e388a3ff53 fix(startup): never panic when the system CA bundle is absent (#6769) 2026-08-28 00:34:51 +00:00
Zhengchao AnandGitHub 22741603f5 test(e2e): finish the helper consolidation onto common.rs (#6766)
- common.rs gains an AdminTransport knob (Signed | Awscurl) with admin_execute_at plus three family wrappers: admin_create_user_via, admin_add_canned_policy_via, admin_attach_user_policy_via; the existing admin_create_user now delegates over the Signed transport.
- Deleted the four signed admin request clones in admin_mfa_test, admin_auth_test, reliant/tiering, and inline_fast_path_cluster_test; each keeps a thin local wrapper over common::admin_request so call sites keep their Option<&str> body shape.
- Deduped the notification_webhook signer onto common::signed_request and the webdav_core signer plus its three admin helpers onto the shared _via helpers.
- Consolidated the S3-client-with-credentials builders: admin_auth s3_client_with, existing_object_tag user_client/sts_session_client, bucket_policy_check create_user_client, and the create_user_s3_client copies in group_delete_test and replication_extension_test now delegate to create_s3_client_with_credentials / build_test_s3_config; replication_extension admin_add_canned_policy and admin_attach_policy_to_user route through the _via helpers on the Signed transport.
- The awscurl-gated suites (existing_object_tag_policy, bucket_policy_check, policy/policy_variables) keep going through the external awscurl binary via AdminTransport::Awscurl, preserving their wire behavior.

Part of rustfs/backlog#1846 (cluster 2).
2026-08-28 00:12:45 +00:00
Zhengchao AnandGitHub 3c89c71f66 fix(s3): round-trip null-version delete-marker identity (#6765)
* fix(s3): round-trip null-version delete-marker identity through listing and delete responses

On a versioning-suspended bucket, a null delete marker's identity was lost on the way back to the client at three points (issue #6745): ListObjectVersions advertised the marker's VersionId as the literal nil UUID instead of null; deleting by that id succeeded but the DeleteObjects/DeleteObject response reported the identity as null with no way to correlate it to the request; and the response lacked DeleteMarker/DeleteMarkerVersionId because the marker-ness comparison mixed the client-facing identity (Some(nil)) with the storage identity (None), so the removal also mis-recorded accounting and fired DeleteMarkerCreated semantics on later paths.

- Listing (bucket_usecase, s3_api/bucket, build_list_versions_next_marker) now maps the synthesized nil UUID to the literal null everywhere it reaches the wire, and VersionMarker::parse folds a nil-UUID marker from older listings into VersionMarker::Null so pagination resumes correctly.
- delete_objects normalizes both sides of the marker-ness comparison via delete_file_info_version_id (matching the adjacent explicit_delete_marker admission check) and reports DeleteMarkerVersionId as null for an explicit null-marker removal.
- resolve_delete_version_state reports delete_marker for an explicit-version delete whose target is a delete marker even when the bucket is versioning-suspended, fixing x-amz-delete-marker on the single-object path.
- The DeleteObjects response entry echoes the version identity the request addressed for marker removals, marker-removal accounting no longer records a marker creation, and notification events fire DeleteMarkerCreated only for actual marker creation.

Fixes #6745

* fix(s3): keep null-marker removal write shape undeleted and report marker semantics response-side

The first cut marked the storage delete request deleted for a null-marker removal, which FileMeta::delete_version interprets as the suspended-bucket delete-mints-a-marker write and re-creates the marker just removed. Carry marker-ness to responses via explicit_delete_removed_marker (single path) and a response-only branch flag (batch path) instead, keeping every storage write shape byte-identical to the pre-fix behavior. Adds an embedded end-to-end regression test covering the full issue #6745 round trip.
2026-08-27 23:55:05 +00:00
Zhengchao AnandGitHub db57fabcbd fix(tier): validate outbound URLs for all warm backend providers (#6764)
WarmBackendS3::new already rejects loopback, private, link-local, and
cloud metadata-service endpoints via validate_outbound_url, but the
Aliyun, Azure, Huaweicloud, Tencent, MinIO, R2, RustFS, and GCS warm
backend constructors built their transition clients directly from
conf.endpoint without the same check.

The endpoint comes from the AddTier admin API, gated only by
SetTierAction, which can be a narrower IAM grant than root. Any
principal holding it could point one of these eight tier types at an
internal address (loopback, RFC1918, link-local, or a cloud metadata
IP) and have the server issue authenticated outbound requests to it, a
server-side SSRF vector that the S3 and Wasabi tier types were already
closed against.

Apply the same validate_outbound_url check at construction time for
all eight providers, before any credentials or network client are
built, mirroring the existing WarmBackendS3 pattern. GCS keeps its
default-endpoint behavior when conf.endpoint is empty and only
validates an explicitly configured endpoint.

Add a regression test per provider asserting that a loopback endpoint
is rejected before any backend/network setup, matching the existing
WarmBackendS3 coverage.

Update the error(format!) ratchet baseline: these are one-shot admin
tier-configuration validation errors returned once per AddTier call,
not per-disk I/O errors that flow through reduce_errs quorum
aggregation (backlog#1845), so the new ::other(format!) call sites do
not introduce a quorum-bucketing hazard. They mirror the pre-existing,
already-baselined warm_backend_s3.rs call site.
2026-08-27 23:37:50 +00:00
Zhengchao AnandGitHub 03888bd266 chore(tier): remove dead trailing_headers config from warm backends (#6762)
TransitionClient::new() in crates/s3-client/src/transition_api.rs computes
trailing_header_support = opts.trailing_headers && override_signer_type == SignatureV4,
but override_signer_type is hardcoded to SignatureDefault at construction
and never mutated afterwards, so the expression is always false regardless
of opts.trailing_headers. The resulting field also has no live reader: its
only reference is inside PutObjectOptions::validate() in
crates/s3-client/src/api_put_object.rs, which is itself
#[allow(dead_code, reason = "MinIO-parity ... no caller in this port")],
and even there the reference to trailing_header_support is commented out.

So trailing_headers: true in the seven warm_backend_*.rs constructors has
never had any effect on request signing or chunked/trailing-header
behavior (stream_sha256 signing is gated separately by
metadata.stream_sha256 && !self.secure). Remove the misleading dead
configuration from the seven provider constructors so it doesn't look
like intentional, load-bearing behavior to future readers.

Found during adversarial self-check while implementing rustfs/backlog#2040 (out of that issue's scope).
2026-08-28 07:35:48 +08:00
Zhengchao AnandGitHub a6a04b5faa refactor(ecstore,rustfs): reuse canonical starts_with_ignore_ascii_case (#6759)
* refactor(ecstore,rustfs): reuse canonical starts_with_ignore_ascii_case

`crates/utils/src/http/metadata_compat.rs` owns the internal metadata key helpers, including `starts_with_ignore_ascii_case`. Two files carried their own byte-identical copies of that predicate: `SetDisks::starts_with_ignore_ascii_case` in ecstore and a free function in the S3 options layer. Both drive internal metadata key classification (`internal_metadata_suffix` and quorum hashing on one side, `should_skip_object_metadata_key` and `is_reserved_user_metadata_key` on the other), so keeping three implementations of one predicate is an avoidable drift risk on a path that decides whether an internal key is treated as user metadata.

Delete both local copies and call the canonical implementation. Every prefix used at these call sites is an ASCII constant or literal, where the canonical byte-slice comparison and the removed `str::get(..n)` form are equivalent; that equivalence was checked differentially over 4.6M (key, prefix) pairs, including keys with multi-byte characters straddling the prefix boundary. No other logic in `internal_metadata_suffix` or `should_skip_object_metadata_key` changed.

Add regression tests on both sides pinning the two properties the switch depends on: internal prefixes match case-insensitively (a mixed-case `X-RustFS-Internal-*` key stays internal), and keys shorter than a prefix never match (they stay ordinary user metadata).

Refs rustfs/backlog#2051

* fix(rustfs): avoid typos-checker false positive in prefix-length test

The test literal "x-rustfs-encryptio" (a deliberate truncation of the
x-rustfs-encryption- prefix, used to assert that a key shorter than every
internal prefix falls through to user metadata) reads as a likely typo of
"encryption" to the repo's typos CI check. Derive it from
RUSTFS_ENCRYPTION_PREFIX via slicing instead of a hand-typed literal, which
both satisfies the linter and ties the truncation to the real constant
instead of a copy-typed guess.

Refs rustfs/backlog#2051
2026-08-28 07:35:09 +08:00
Zhengchao AnandGitHub 18068eb7e5 fix(deps): move off the yanked chacha20 0.10.1 (#6768)
chacha20 0.10.1 was yanked on crates.io today, which fails the Cargo Deny gate (error[yanked]) on every branch. cargo update -p chacha20 to 0.10.2; no API change, all dependents are semver-compatible.

Verification: cargo check -p rustfs-crypto; the Cargo Deny job on this PR is the authoritative gate.
2026-08-28 07:03:24 +08:00
Zhengchao AnandGitHub 921a48bd14 refactor(lifecycle): reuse the replication tag parser (#6761)
`crates/lifecycle/src/tagging.rs` carried a byte-identical copy of the `form_urlencoded` tag decoder already owned by `rustfs-replication`, plus a duplicate of its test. Since `crates/lifecycle` already depends on `rustfs-replication`, replace the copy with a `pub(crate) use` re-export: no new crate edge, one parser, and no second implementation to drift from the replication contract. The `rule.rs` call site is unchanged.

Also drop `crates/ecstore/src/bucket/lifecycle/tagging_boundary.rs`, a migration-era boundary shim with zero call sites in the tree.
2026-08-27 22:56:13 +00:00
Zhengchao AnandGitHub 2e6511566e refactor(rustfs): consolidate bucket metadata import match arms (#6760)
The import_bucket_metadata handler carried eight match arms whose bodies were
byte-identical apart from the type a payload is validated against and the pair
of BucketMetadata fields it lands in, so every arm repeated the same warn! call
and the same metadata lookup. Fold them into one or-pattern arm backed by
apply_imported_bucket_config, where a single conf_name match owns both the
validated type and the destination field pair and can no longer drift apart.

Validation still runs before the metadata lookup, the warn! event, fields, and
label are unchanged, and the BUCKET_POLICY_CONFIG and BUCKET_QUOTA_CONFIG_FILE
arms keep their own handling. Regression tests drive the full mapping table:
each config file's payload lands only in the field it owns, an unparsable
payload leaves the field untouched, and a rejected entry does not stop the
remaining ones from being imported.

Refs rustfs/backlog#2052
2026-08-27 22:55:09 +00:00
Zhengchao AnandGitHub ec8abb19ab refactor(s3-client): reuse rustfs-utils header classification instead of duplicating it (#6758)
crates/s3-client/src/utils.rs carried a verbatim copy of the header
classification tables and predicates owned by
crates/utils/src/http/headers.rs: SUPPORTED_HEADERS (same 11 keys),
SUPPORTED_QUERY_VALUES (same 9 keys), and is_standard_header /
is_storageclass_header / is_amz_header / is_rustfs_header /
is_minio_header with byte-identical bodies. The duplication was already
half-resolved and inconsistent — the local is_amz_header called
rustfs_utils::http::is_sse_header while consulting its own tables — and
s3-client already depends on rustfs-utils with the "full" feature, so
reusing the canonical owner adds no crate edge.

The sole caller, PutObjectOptions::header(), now imports the five
predicates from rustfs_utils::http. Semantics are unchanged: both sides
normalize with to_lowercase(), return false for unknown keys, and the
storage-class constants are the same string ("x-amz-storage-class" from
s3s::header::X_AMZ_STORAGE_CLASS vs rustfs_utils AMZ_STORAGE_CLASS), so
the set of user-metadata headers passed through verbatim rather than
prefixed with x-amz-meta- is identical.

SUPPORTED_QUERY_VALUES is deleted outright: s3-client had no reader for
it (utils consumes its own copy via is_standard_query_value). The
base64_encode/base64_decode helpers and their rustfs/rustfs#4811
regression test stay untouched, and lazy_static remains a dependency
because crates/s3-client/src/constants.rs still uses it.

Refs rustfs/backlog#2050
2026-08-28 06:53:15 +08:00
Zhengchao AnandGitHub 2a8be5566d refactor(concurrency): consolidate ForegroundPressure into workload owner (#6757)
ForegroundPressure had two definitions with byte-identical pressure computation: one in ecstore data-movement backpressure and one in the heal manager queue. That duplication is a violation of the ARCHITECTURE.md invariant that each type has exactly one definition, and it means any future change to the utilization math has to land twice.

Add the canonical `ForegroundPressure` and a `foreground_pressure(snapshot, read_threshold_pct, write_threshold_pct)` function to `crates/concurrency/src/workload.rs`, which already owns `WorkloadClass`, `AdmissionState`, and the admission snapshot contract. Both existing consumers already depend on `rustfs-concurrency`, so no crate edge is added.

The filter_map pipeline is transferred verbatim, preserving all five boundary behaviors (zero threshold, zero limit, missing entry, missing active count, and the `Saturated` full-utilization special case), the mul-before-div percentage normalization, the `>=` threshold comparison, and the read-then-write ordering that makes `max_by_key` break utilization ties toward the write class. The enable switch is deliberately left out: ecstore gates on `config.enabled` while heal gates on `mainline_throttle_enable` plus a both-thresholds-zero check, so each call site keeps its own condition.

This is the expand step only. The ecstore and heal copies are untouched and are removed by the follow-up migrate task.

Refs rustfs/backlog#2047
2026-08-28 06:52:50 +08:00
Zhengchao AnandGitHub bcbc58b6a0 refactor(ecstore): extract shared warm backend S3 constructor (#6755)
The seven S3-compatible warm backend providers (Aliyun, Azure, Huaweicloud, Tencent, MinIO, R2, RustFS) each carry a byte-identical copy of the same statically-credentialed TransitionClient construction and of the same optimal_part_size helper. Add both to the module that already owns the WarmBackend trait and WarmBackendS3, so the per-provider migrate step can drop its duplicate without redesigning anything.

bucket_lookup is a parameter rather than a constant because the providers split into two families: Aliyun, Azure, Huaweicloud, and Tencent pin BucketLookupDNS, while MinIO, R2, and RustFS leave it at the BucketLookupAuto default. Hardcoding either value would silently change bucket addressing for the other family during the migrate step.

Error texts, validation order, prefix and host/port normalization are reproduced exactly from the Aliyun/MinIO family. No provider file is touched and no production caller exists yet, so the new unit tests are the first callers.

Refs rustfs/backlog#2040
2026-08-27 22:16:47 +00:00
Zhengchao AnandGitHub 28fa412a06 refactor(ecstore): extract shared audit/notify KVS table constructors (#6756)
The audit and notify subsystems each declare their own default KVS table for the same nine delivery targets. For amqp, nats, pulsar, postgres and kafka the two declarations are byte-identical; for redis and mysql they differ only in a single default literal (the pub/sub channel and the destination table). Keeping two copies means every default or key-order change has to be made twice, and a missed edit silently changes what admin config reports for one subsystem only.

Add `config::target_defaults` with one constructor per shared table, taking the diverging literal as a parameter for redis and mysql, plus a small `kv` helper that replaces the repeated `KV { .. }` literals. Key order is reproduced exactly because it drives the order the admin API lists keys in. Unit tests pin the full ordered key/value/hidden_if_empty triple of every table against hard-coded literals, and cover both the audit and the notify literal for the two parameterized tables.

Webhook and mqtt are deliberately left out: audit's webhook table carries extra batching and retry keys, both webhook tables disagree on key order and on the auth-token hidden_if_empty flag, and mqtt disagrees on qos, keep-alive interval and reconnect interval. Those are real behavioral forks, not duplication, so they stay declared in place.

This is the expand step only. Nothing calls the new module yet, so audit.rs and notify.rs are untouched and no default changes; the constructors carry an item-level allow(dead_code) until the migrate step points both files at them.

Refs rustfs/backlog#2044
2026-08-27 22:16:03 +00:00
Zhengchao AnandGitHub 95c926dc79 ci(release): delete preview releases after the deliverable publishes (#6729)
Preview tags stay as the traceability record for the validated commit, but their GitHub Releases are internal validation state and should not accumulate on the Releases page next to real deliverables.

Add a cleanup-preview-releases job that runs after publish-release succeeds for a release or prerelease tag and deletes every Release whose tag is exactly <target>-preview.<digits>. The tags themselves are kept: the job never passes --cleanup-tag. Tag matching uses jq string operations rather than a regex over the version, so dots in the version cannot widen the match, and the release listing is fetched before filtering so an API failure aborts the job instead of looking like there was nothing to clean up.

Extend the preview release workflow guard with the job condition, the delete invocation, both tag-matching filters, and an absent check for --cleanup-tag.
2026-08-27 16:18:05 +08:00
Zhengchao AnandGitHub 7b17d46ca9 refactor(site-replication): move business tests next to the service module (#6716)
* refactor(site-replication): move business tests next to the service module

backlog#1840 PR5: 79 business-logic tests (plus 12 helpers, 6 of them small fixtures kept on both sides) move from the admin handler file's test module into rustfs/src/site_replication/tests.rs, next to the code they exercise: peer connection/TLS/DNS/egress validation, the peer client cache and payload wire contract, retry-queue classification/settlement/escalation/backoff, the repair state machine, bootstrap-plan construction, lifecycle expiry subsetting, bucket-target reconciliation, endpoint/identity normalization, and state serialization. The 149 tests that exercise the admin handlers, apply/reconcile paths, status/resync builders, and the four include_str! tripwires stay in rustfs/src/admin/handlers/site_replication.rs with their subjects (229 total conserved: 149 + 79 + 1).

The issue's PR5 also called for converting the source-order tripwire at the old file's line 11339 into a behavior test; both adversarial review passes re-derived all four tripwires against the shrunken file and found them non-vacuous and byte-identical in the regions they guard (the handler bodies, which did not move), so they stay as source-text assertions.

Supporting changes: the root facade's site_replication consumer gains cfg(test) re-exports (endpoint types, merge_incoming_replication_config, five lifecycle DTO types) so the relocated tests stay off the direct s3s/admin surfaces — including rewriting the one inline crate::admin BucketMetadata path a moved test carried over (review finding); tests.rs joins the logging-guardrail checked list; the embedded-secrets guard comment follows the validate_peer_connection_inner fixtures to their new file.

Verified: cargo check -p rustfs --all-targets clean; cargo nextest run -p rustfs --lib 3856/3856 passed; relocated tests run under site_replication::tests::; make pre-commit green including the s3s footprint ratchet; logging and embedded-secrets guards green.

Refs rustfs/backlog#1840

* style(site-replication): apply rustfmt import ordering
2026-08-27 15:17:16 +08:00
Zhengchao AnandGitHub 4bbc1d5640 test(ecstore): wait for multipart rename tail epochs (#6723) 2026-08-27 05:28:55 +00:00
Zhengchao AnandGitHub 8ddbf05924 test(targets,notify,audit): share a builder MockTarget testkit (#6717)
* test(targets): ship a builder MockTarget testkit and retire the in-crate Target mocks

Adds crates/targets/src/testkit.rs with a builder-style MockTarget implementing Target<E> for every E: PluginEvent, with orthogonal off-by-default knobs: disabled/active override, health delay plus health-started signal plus a drop-guard counter proving a cancelled probe future was dropped, an init failure budget (usize::MAX = always fail) plus blocking init plus an init counter, a close counter/signal/semaphore gate with a runtime block toggle, a save counter plus save failure budget, caller-supplied store and failed-store handles, and a shared final-failure counter. Clones and clone_dyn share all counters, so an observer clone keeps watching a target after it is boxed into a runtime.

The module is gated as #[cfg(any(test, feature = "test-support"))]: in-crate unit tests get it via cfg(test), and downstream test suites opt in through the new off-by-default test-support cargo feature (test-support = [], activating no dependencies).

Migrates the five in-crate duplicate mocks onto it: the plugin.rs registry-factory TestTarget, the runtime/adapter.rs lifecycle TestTarget (init/close/store knobs), the runtime/mod.rs TestTarget plus HealthDropGuard (close gating and health-probe tests), the target/mod.rs MoveTestTarget (folded into the test_support helper constructors used by the NATS JetStream failed-store tests), and the target/mod.rs StoreBackedTarget (the default send_from_store purge test; the mock deliberately does not override send_from_store or handle_terminal_failure). The forced init failure now uses TargetError::Initialization instead of the old adapter mock's Configuration; the adapter derives its redacted failure summary from the target id alone, so the migrated assertions are unchanged.

Leak guard: testkit unit tests assert the crate manifest still declares default = [] and that test-support = [] stays a pure cfg gate, complementing the compile-level cfg gate that keeps the mock out of production builds.

Part of rustfs/backlog#1846 (cluster 3, step 1).

* test(notify,audit): migrate the Target mocks onto the shared testkit

Retires the hand-written Target mocks in crates/notify and crates/audit in favor of rustfs_targets::testkit::MockTarget: notify's notifier.rs TestTarget/DeferredTestTarget/ClosableTestTarget, lifecycle.rs BlockingInitTarget/RetryInitTarget (rebuilt as observed MockTarget templates cloned by their plugin-descriptor factories, sharing the init signal, close counter, and single-failure init budget across generations), runtime_view.rs TestTarget, runtime_facade.rs TestTarget, and audit's pipeline.rs MockTarget, system.rs TestTarget, registry.rs CloseTestTarget, plus TestTarget and FailingTarget in audit/tests/pipeline_layer_test.rs. Removing the two integration-test mocks also removes their respelled PluginEvent bound (E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned), which the plugin-contract rules require to be spelled only via PluginEvent.

lifecycle.rs ReplayTarget stays bespoke on purpose: its generation tags, mpsc observation channels, gated send_raw delivery, and ObservedQueueStore model the replay pipeline itself and would contort a general-purpose mock. The other bespoke mocks named out of scope in PR-3a (ProgrammedTarget, ClassifyingTarget, the ReloadableTargetTls fakes) are likewise untouched.

New testkit knobs, each defaulted off and unit-tested: with_id (rename a clone while keeping the shared counters, for factory templates), with_first_save_gate (the first save notifies entered and waits on release; several mocks may share one pair), with_health_gate (is_active waits on a release handle after notifying health_started), with_delivery_snapshot (fixed snapshot overriding the store-derived default), with_close_failures (close-failure budget, default TargetError::Storage) with with_close_failure_error to shape the variant (audit's registry test pins TargetError::Unknown), and an always-on is_enabled call counter exposed as enabled_call_count (notifier's generation tests count dispatcher selections through it).

Both crates enable the testkit through a dev-dependency on rustfs-targets with the test-support feature; the feature stays out of default and activates no dependencies, so production builds are unchanged.

Part of rustfs/backlog#1846 (cluster 3, step 2).
2026-08-27 11:28:16 +08:00
Zhengchao AnandGitHub 0e27f57c40 fix(test): wait for EC materialization in relocated-pool resume fixture (#6718)
fix(test): wait for EC write materialization before staging relocated-pool fixture

An erasure-coded write returns once write-quorum disks commit, so a lagging disk can legally still be missing its xl.meta when the relocated-pool resume test starts staging its fixture by iterating every disk of the owning pool. Under CI load this raced into a NotFound panic in the staging loop. Add a bounded readiness poll that waits for xl.meta on every pool disk before the normalization and staging steps.

Fixes #6703
2026-08-27 11:26:20 +08:00
Zhengchao AnandGitHub e760d44e65 chore(rustfs): remove stale manual-test-runners feature and README (#6715) 2026-08-27 10:24:03 +08:00
Zhengchao AnandGitHub 9e27ede8f0 test(ci): point quarantine machinery at the legs it must protect (#6714) 2026-08-27 10:23:45 +08:00
Zhengchao AnandGitHub 5c40570edd refactor(site-replication): migrate call sites to PeerAdminRequest, drop wrappers (#6713) 2026-08-27 10:23:27 +08:00
Zhengchao AnandGitHub a6c5d80069 test(rustfs): move manual bench tools to examples, gate Swift suites (#6712) 2026-08-27 10:23:14 +08:00
Zhengchao AnandGitHub f1f8057154 fix(ecstore): tighten decommission test-helper cfg gates to test-util (#6708) 2026-08-27 10:08:41 +08:00
Zhengchao AnandGitHub 7b7e5f38f7 test(get): tolerate quorum-tolerated disk gaps in relocated-pool test (#6707) 2026-08-27 10:08:25 +08:00
Zhengchao AnandGitHub 52703e0da6 docs(rio): record the checksum hasher verdict and pin shared vectors (#6706) 2026-08-27 10:08:09 +08:00
Zhengchao AnandGitHub 6ab14981ba refactor(site-replication): converge send_peer_* into one request builder (#6705) 2026-08-27 10:07:56 +08:00
Zhengchao AnandGitHub 12b5e69a7b test(ci): quarantine the relocated-pool GET resume fixture flake (#6704) 2026-08-27 10:07:43 +08:00
Zhengchao AnandGitHub c00e5491b5 refactor(s3-client): drop superseded per-algorithm checksum plumbing (#6700)
refactor(s3-client): remove the superseded per-algorithm checksum plumbing

Deletes the write-only RequestMetadata.add_crc pipeline (assigned but never read since the port), the dead MinIO-parity Checksum constructors and CompletePart accessor, and key_capitalized (identical to key). The five hand-rolled x-amz-checksum-* response-header if-lets in the streaming and multipart paths collapse into one checksum_header_value helper, ChecksumMode's inherent to_string becomes a Display impl, and checksum.rs drops its file-wide allow blanket now that the file is lint-clean.

Refs rustfs/backlog#1844 (PR2 of 3).
2026-08-27 01:06:04 +00:00
Zhengchao AnandGitHub a6ea4ac8f3 refactor(admin): move site-replication service core out of handlers (#6699)
Mechanical move-only extraction for backlog#1840 PR1+PR4: the site-replication state (load/parse/persist/RMW transaction), repair state machine, peer transport (client cache, DNS resolver, send_peer_* family), retry queue, and the four storage-side hooks move from rustfs/src/admin/handlers/site_replication.rs into the new infra-layer module rustfs/src/site_replication/ ({mod,state,state_lock,identity,transport,retry,repair,hooks}.rs). The admin handler file keeps route registration, all Operation impls, request/response glue, and the in-file test module, and re-exports the moved items so existing paths keep resolving. admin/site_replication_identity.rs and admin/site_replication_state.rs relocate wholesale as identity.rs/state_lock.rs.

Storage access from the moved code goes through a new site_replication consumer module in the root facade (rustfs/src/storage_api.rs), including an s3 shim so the module stays off the direct s3s surface (file count stays at the 215 baseline). The three admin runtime-source wrappers the moved code needs (outbound TLS generation incl. the test atomic, outbound TLS state, runtime port) are reproduced locally; the TLS-generation trio moves out of admin/runtime_sources.rs since site replication was its only consumer. The one non-verbatim rewrite: site_replication_peer_payload inlines encrypt_stream_io in its encrypted branch, which is provably the branch encode_compatible_admin_payload always took for the /minio/admin peer-join wire path.

app/bucket_usecase.rs now imports the three bucket hooks from crate::site_replication, deleting the three app->interface entries from the layer baseline (shrink-only). The peer-client cache test moves with the owner-local SITE_REPLICATION_PEER_CLIENT static into transport.rs (228+1 = 229 tests conserved). New module files are added to the logging-guardrail checked list; the s3_error! line baseline tightens 1620 -> 1619; global-state/config-consumer inventories and ARCHITECTURE.md pointers updated.

Verified: cargo check -p rustfs --all-targets clean; cargo clippy --workspace --all-targets clean; cargo nextest run -p rustfs --lib 3852/3852 passed; make pre-commit green; scripts/check_layer_dependencies.sh green with baseline-only deletions; line-multiset conservation audit over the moved code accounts for every non-verbatim line (visibility bumps, import rewrites, fmt reflow).

Refs rustfs/backlog#1840
2026-08-27 09:01:11 +08:00
Zhengchao AnandGitHub dda841d8de refactor(ecstore): retire set_disk lint blankets via explicit imports (#6697)
refactor(ecstore): retire the set_disk lint blankets by making the prelude explicit

backlog#1823 step 1 / backlog#2029 road 2. Removes the last two module-level lint blankets in ecstore: set_disk/mod.rs #![allow(unused_imports)] and #![allow(unused_variables)], restoring both lints for the whole 40K-line subtree, and deletes the register line for the unused_variables blanket in the same diff (the guard from #6155 is a bidirectional exact match).

The unused_imports blanket existed because 14 submodules consumed mod.rs as a glob prelude (use super::* / use super::super::*), and rustc does not track consumption through glob re-exports. Each glob is now an explicit use super::{...} list, keeping mod.rs as the single import hub while making every import lint-checkable. Names consumed only by test or test-util units carry #[cfg(test)] / #[cfg(all(test, feature = "test-util"))] / #[cfg(any(test, feature = "test-util"))] gates matching their consumers; storage-api traits are routed through the storage_api_contracts facade per the architecture guard.

The sweep then deleted the genuinely dead imports the blanket was hiding (chrono::Utc, glob::Pattern, futures::task::AtomicWaker, rustfs_lock LocalLock, AsyncBatchProcessor, rand::Rng, std::future::Future among others in mod.rs, plus stale scoped imports and one empty test module shell across the subtree). One unused_variables finding surfaced: flush_read_version_coalescer_pending's lane_key is read only by the #[cfg(test)] counter block, handled with the cfg(not(test)) let _ pattern established in #6158.

Verification: cargo check zero warnings versus the 9cf276ed2 baseline on five lanes (default lib / --tests / rio-v2 --tests / test-util --tests / test-util,rio-v2 --tests; the --tests lane keeps the same three pre-existing core/pools.rs and store/object.rs dead-code warnings main already has); clippy --lib --tests -D warnings clean with test-util,rio-v2; cargo nextest run 4567 passed; make pre-commit exit 0.
2026-08-27 08:01:18 +08:00
Zhengchao AnandGitHub 09ec797a66 refactor(checksums): unify s3-client checksum dispatch in one registry (#6696)
The s3-client ChecksumMode previously duplicated per-algorithm header names, wire names, digest lengths, and checksum-type capability tables in EnumSet-mask matches. ChecksumAlgorithm in rustfs-checksums now owns that metadata behind exhaustive matches (a new variant fails to compile until its metadata is decided), and ChecksumMode delegates through a single algorithm() bridge. Wire behaviour is pinned unchanged by tests on both sides.

Refs rustfs/backlog#1844 (PR1 of 3).
2026-08-27 08:01:08 +08:00
Zhengchao AnandGitHub a169dd01a6 chore(release): prepare 1.0.0-rc.4
* chore(release): prepare 1.0.0-rc.4

* chore(release): align release assets for 1.0.0-rc.4
2026-08-27 07:16:52 +08:00
Zhengchao AnandGitHub 64739778c4 refactor(admin): consolidate json_response, empty_response, and extract_query_params into admin utils (#6694)
The admin surface had accumulated one near-identical response helper per handler file. This folds the byte-equivalent ones into `rustfs/src/admin/utils.rs` so the wire shape of an admin JSON answer is pinned in one place instead of being re-derived twelve times.

Folded into `crate::admin::utils`:

- `json_response(status, &value)` — 9 local definitions removed: batch_job.rs, kms_backup.rs, oidc.rs, diagnostics.rs (identical signature), object_data_cache.rs and site_replication.rs (hard-coded `StatusCode::OK`, whose call sites now pass `StatusCode::OK` explicitly), ilm_transition.rs (arguments were `(&value, status)` and are swapped at every call site), and kms_key_metadata.rs / kms_key_lifecycle.rs (concrete response types now covered by the generic helper).
- `empty_response(status)` — 2 local definitions removed: site_replication.rs (`Body::empty()`) and table_catalog/mod.rs (`Body::default()`); `Body::empty()` is defined as `Body::default()`, so the two were already the same response.
- `extract_query_params(uri)` — 4 local definitions removed: kms_keys.rs (was `pub(super)`), replication.rs, batch_job.rs, config_admin.rs. All four bodies were behaviourally identical (`form_urlencoded::parse` over `uri.query()`, last-wins on repeated keys, valueless parameters kept as empty strings); they differed only in blank lines. kms_key_lifecycle.rs, which imported the kms_keys copy, now imports the shared one.

Intentionally left alone:

- heal.rs `json_response` — different shape: returns a bare `S3Response` (not `S3Result`) and additionally sets `CONTENT_LENGTH`.
- kms_rekey.rs `json_response` — same divergent shape as heal.rs: bare `S3Response` over already-serialized `Vec<u8>`.
- idp_compat.rs `json_response` — encrypts the payload via `encode_compatible_admin_payload`; it is not a duplicate of the plain JSON helper.
- scanner.rs `json_response` — takes raw `Vec<u8>`, and `ScannerCycleStateResetHandler` genuinely passes a byte literal rather than a serializable value, so the local helper stays.
- oidc.rs `extract_query_param` — singular, returns `Option<String>` for one key, hand-rolls its own splitting via the `urlencoding` crate; a different function, not a variant of the map builder.

Wire behaviour on the success path is byte-identical everywhere: same status, same `Content-Type: application/json` (every local copy spelled the same value, whether via a per-file `JSON_CONTENT_TYPE`/`CONTENT_TYPE_JSON` constant, `HeaderValue::from_static`, or `"application/json".parse()`), same serialized body bytes, and no other header. The only behavioural change is the message text on the serde-serialization-failure arm, which is now uniformly `failed to serialize response: {e}`; that arm is unreachable for these owned response structs and the acceptance criteria pin only status and content type.

No `include_str!` self-grep assertion needed updating: the affected tests in ilm_transition.rs, site_replication.rs, kms_keys.rs, kms_key_metadata.rs, kms_key_lifecycle.rs, object_data_cache.rs, and table_catalog/tests.rs are all bounded by handler `impl Operation` / entry-point markers that sit well after the removed helpers, and none of them assert on a `json_response`, `empty_response`, or `extract_query_params` string.

Tests: `rustfs/src/admin/utils.rs` gains `json_response_carries_status_content_type_and_serialized_body`, `json_response_reports_serialization_failure_as_internal_error`, `empty_response_has_no_body_and_no_headers`, `extract_query_params_decodes_percent_escapes`, and `extract_query_params_keeps_valueless_parameters_and_survives_no_query`. The percent-decoding coverage previously in batch_job's `extract_query_params_decodes_job_id` moves there, and batch_job keeps its own end-to-end coverage as `require_job_id_decodes_and_rejects_missing_and_empty`.

Reference: rustfs/backlog#1829 T6
2026-08-26 21:57:13 +00:00
Zhengchao AnandGitHub 030a87013c fix(e2e): implement scanner lease RPC stubs (#6693) 2026-08-26 20:42:29 +00:00
Zhengchao AnandGitHub 94a61d789a fix(guards): catch dotted-form leaf deps, pin madmin to rustfs-signer (#6692)
fix(guards): catch dotted-form leaf deps; pin madmin to rustfs-signer
2026-08-27 03:49:29 +08:00
Zhengchao AnandGitHub 31031f2a46 fix(ci): bound cold ILM compilation (#6689) 2026-08-27 03:17:01 +08:00
Zhengchao AnandGitHub 61914ac4ad refactor(admin): route tier, bucket metadata, archive, transition, and oidc auth through the shared gate (#6688) 2026-08-27 03:16:44 +08:00
Zhengchao AnandGitHub 0e92eac2c2 refactor(admin): route pool, rebalance, and system authorization through the shared gate (#6687) 2026-08-27 03:16:31 +08:00
Zhengchao AnandGitHub 3699b6b88d refactor(admin): route user management authorization through the shared gate (#6686) 2026-08-27 03:16:06 +08:00
Zhengchao AnandGitHub 4154a3b7ca refactor(admin): route IAM policy and group auth through the shared gate (#6685) 2026-08-27 03:15:50 +08:00
Zhengchao AnandGitHub a42046b79c feat(rpc): dual-write a typed not-initialized code on control-plane responses (#6684) 2026-08-27 03:15:32 +08:00
Zhengchao AnandGitHub f4cc919401 docs(architecture): adjudicate the io-metrics leaf dependency on the s3-ops contract crate (#6683) 2026-08-27 03:15:07 +08:00
Zhengchao AnandGitHub 8f7ccab4ed fix(ci): make s3s footprint ratchet count the tree, not stdin (#6681)
fix(ci): give the s3s footprint ratchet explicit rg paths so CI counts the tree, not stdin
2026-08-26 22:50:15 +08:00
Zhengchao AnandGitHub 78935c34c9 refactor(common): drop transitional heal/scanner contract shims (#6680)
Every consumer now imports rustfs-heal-contracts / rustfs-scanner-contracts
directly and rg 'rustfs_common::(metrics|heal_channel|last_minute)' reports
zero hits, so the backlog#1843 re-export shims and the transitional
rustfs-common -> contracts dependency edges can go. rustfs-common no longer
recompiles on scanner/heal type changes. Doc references to the moved files
follow the new paths.
2026-08-26 22:49:55 +08:00
Zhengchao AnandGitHub fda93adafe fix(ecstore): migrate test-only client import (#6678) 2026-08-26 22:26:02 +08:00
Zhengchao AnandGitHub 38e93f553d style(rustfs): format object split guard (#6676) 2026-08-26 22:11:27 +08:00
Zhengchao AnandGitHub 41d7db6d57 test(connect): match offline enrollment rejection (#6675)
test(connect): match offline rejection message
2026-08-26 22:05:10 +08:00
Zhengchao AnandGitHub 57b0a136d5 test(ci): serialize rekey Vault e2e (#6673) 2026-08-26 22:04:44 +08:00
Zhengchao AnandGitHub 2ebf8bc138 refactor(ecstore): drop the client shim, import rustfs-s3-client directly (#6668)
* refactor(ecstore): drop the client shim, import rustfs-s3-client directly

Completes the migration window opened by the rustfs-s3-client extraction (rustfs/backlog#1842 PR3): every consumer now imports the client crate directly and the crate::client shim is deleted.

- All in-crate crate::client:: paths (tier warm backends, tier core, lifecycle tier_sweeper, replication storage boundary, set_disk) now import rustfs_s3_client::* directly; crates/ecstore/src/client/mod.rs and the lib.rs mod client declaration are gone.
- The two server-side modules historically misfiled under client/ move to their real homes: object_api_utils.rs to crates/ecstore/src/object_api/ (it builds engine-side object readers/writers), and object_handlers_common.rs to crates/ecstore/src/bucket/lifecycle/ (it is the lifecycle noncurrent-version cleanup helper). The latter now routes its replication calls through the lifecycle replication_sink boundary (schedule_delete wrapper and the sink's ReplicationObjectBridge re-export), as the lifecycle guard requires.
- The ecstore public facade drops api::client: object_api_utils is exposed as api::object_api_utils, and the rustfs crate takes admin_handler_utils (AdminError) from rustfs-s3-client directly (new dependency).
- Guard updates: the migration guard no longer pins mod client in ecstore's lib.rs or the admin_handler_utils facade module (it pins the new api::object_api_utils facade instead), and the module-lint register follows object_api_utils.rs to its new path.

Verification: cargo check -p rustfs-ecstore --all-targets and -p rustfs; cargo fmt --all; tier/transition/lifecycle-focused nextest (626 passed) and the decommission/rebalance/heal families in a filtered run (603 passed; the full-suite parallel run only fails on this machine's known decommission/rebalance baseline flakes, which pass in filtered reruns and fail identically on pristine origin/main); layer/migration/s3s/logging/error-format/doc-path guard scripts all pass.

* docs(architecture): record the S3 client extraction and reword invariant 4 (#6669)

Closes the documentation step of rustfs/backlog#1842. ARCHITECTURE.md invariant 4 now states the serving-vs-consuming distinction the adversarial ruling asked for: ecstore must not serve HTTP/S3 wire types, while consuming remote S3 endpoints is a legitimate engine capability that lives in the extracted rustfs-s3-client crate. The violation note is updated from the pre-extraction snapshot (58 files, embedded client) to the current ratcheted state (shrink-only S3S_ECSTORE_FILES_BASELINE in scripts/check_s3s_footprint.sh, object_lock converted first), and the crate map gains s3-client. ecstore-module-split-plan.md gets the client-directory entry the plan was missing: a Current Shape row and a completed-extraction section describing the pure-move + shim + direct-import sequence and the re-homing of the two misfiled server-side modules.
2026-08-26 22:02:36 +08:00
Zhengchao AnandGitHub 0e56ef4f1c refactor(rustfs): split object_usecase.rs into per-operation app/object modules (#6670)
* refactor(rustfs): carve app/object out of object_usecase.rs — shared, extract, test_support children (backlog#1841 step 1)

Mechanical move-only split of rustfs/src/app/object_usecase.rs (19.7K lines). The file body moves to rustfs/src/app/object/mod.rs, and the first self-contained slices move into children: shared.rs (cross-cutting helpers: quota admission, response checksum injection, object-lock write validation, table-catalog mutation guard, deadlock request guard, proxy passthrough utilities), extract.rs (snowball auto-extract path incl. tar/pax helpers and execute_put_object_extract), and cfg(test) test_support.rs for cross-module test scaffolding. object_usecase.rs stays as a thin pub use facade so every existing crate::app::object_usecase:: path keeps working.

No behavior change: items move verbatim; the only source edits are visibility widenings required by the new module boundaries (private -> pub(super); pub(super) -> pub(crate) for the three helpers multipart_usecase and the app gating tests import). Guard scripts that pinned rustfs/src/app/object_usecase.rs now scan the rustfs/src/app/object tree, and the table_catalog source-text guard test concatenates the split files.

* refactor(rustfs): move the GetObject read path into app/object/get.rs (backlog#1841 step 2)

Move-only continuation of the object_usecase split: cold-fill orchestration, disk-permit admission, streaming readers and resume control, stream-buffer tuning, execute_get_object / execute_get_object_attributes, the GET replication proxy helpers, and their unit tests move from app/object/mod.rs into app/object/get.rs. Items keep their original text; cross-module call sites rely on the visibility widenings introduced in step 1.

* refactor(rustfs): move the PutObject and CopyObject paths into app/object (backlog#1841 step 3)

Move-only continuation: put.rs takes the PUT body admission and timeout readers, zero-copy and eager-commit machinery, execute_put_object, and the PUT unit tests; copy.rs takes the copy namespace/lifecycle lock helpers and execute_copy_object with its tests. Two source edits beyond visibility widenings: PutObjectChecksums fields become pub(super) (read by shared::apply_trailing_checksums across the new module boundary) and one relative super::storage_api call in the copy path becomes crate::app::storage_api since super now resolves to app::object. The table_catalog source-text guard concatenates the new files.

* refactor(rustfs): finish the object_usecase split — delete, head, restore modules (backlog#1841 step 4)

Move-only completion: delete.rs takes the delete helpers, cfg(test) delete hooks, and execute_delete_object/execute_delete_objects; head.rs takes execute_head_object with the HEAD replication proxy helpers; restore.rs takes execute_restore_object. app/object/mod.rs is now just the shared import prelude, module wiring, and the DefaultObjectUsecase struct with its constructors, accessors, and the execute_select_object_content delegation; the emptied tests module is gone. The delete re-export glob is cfg(test)-gated because its only cross-module consumers are the delete test hooks.

The table_catalog source-text guard now isolates the delete entrypoints from app/object/delete.rs, and doc/comment references that pointed at rustfs/src/app/object_usecase.rs internals now point at the per-operation modules.
2026-08-26 22:02:11 +08:00
Zhengchao AnandGitHub 9f245e3fd4 refactor(ecstore): move object_lock WORM evaluation onto storage-level types (#6666)
The object_lock module evaluated WORM state through s3s wire DTOs (ObjectLockRetention, ObjectLockLegalHold, DefaultRetention, Date) and s3s header constants, keeping the storage engine coupled to the serving protocol (rustfs/backlog#1842, ARCHITECTURE.md invariant 4). This PR gives the module its own storage-level vocabulary and pushes the DTO conversions to the boundaries that already speak s3s.

New crates/ecstore/src/bucket/object_lock/types.rs defines RetentionMode, LegalHoldStatus, ObjectRetention, ObjectLegalHold, and DefaultRetention with no s3s dependency. objectlock.rs parses persisted metadata into these types using the rustfs-utils lowercase header constants (the same literal keys as before, pinned by the existing g-key-002 test). objectlock_sys.rs evaluates retention/legal-hold/default-retention from them; the fail-closed error messages and decision logic are unchanged line for line where possible.

Boundary conversions:
- bucket/metadata_sys.rs gains default_retention_from_object_lock_config, converting the persisted s3s configuration into the storage-level DefaultRetention; a rule without a usable GOVERNANCE/COMPLIANCE mode converts to None exactly like the evaluation code always ignored it, and days/years pass through so an invalid period still fails closed at evaluation time.
- check_object_lock_for_deletion_with_config becomes check_object_lock_for_deletion_with_default_retention (it only ever read the default retention); the lifecycle object_lock_boundary keeps the old s3s-typed signature and converts.
- The ObjectLockApi / ObjectLockStatusExt trait impls for the s3s DTOs move next to the persisted configuration owner in bucket/metadata.rs; the traits stay in object_lock/mod.rs.
- check_retention_for_modification now takes Option<RetentionMode>. The serving-layer wrappers (rustfs storage_api, set_disk options path) convert the request string with the new RetentionMode::parse_exact, which accepts only the canonical spelling — preserving the historical literal comparison where a non-canonical requested mode reads as a mode change and stays blocked.
- rustfs app-layer wrappers return the storage types; the replication-overwrite gate in object_usecase.rs uses the typed API (legal_hold.is_on(), RetentionMode::Compliance).

Ratchet: the ecstore-scoped s3s counter drops 42 -> 39 and the repo-wide file counter 211 -> 208 in scripts/check_s3s_footprint.sh.

Verification: cargo check -p rustfs-ecstore --all-targets and -p rustfs (lib+bins); cargo clippy -p rustfs-ecstore --all-targets and -p rustfs --lib --bins (clean); cargo nextest run -p rustfs-ecstore --no-fail-fast (4534/4542; the 8 failures are the same store::rebalance / store::heal machine-baseline set that fails identically on pristine origin/main, plus one fencing flake that passes in isolation); all object_lock/retention/legal-hold tests pass; guard scripts (layer deps, migration rules, s3s footprint, logging, error-format ratchet, doc paths) pass.
2026-08-26 21:25:35 +08:00
Zhengchao AnandGitHub 62a767a52d test(connect): add short credential E2E profile (#6664)
* feat(connect): add short credential E2E profile

* ci(connect): test short credential boundary
2026-08-26 21:25:02 +08:00
Zhengchao AnandGitHub a96dd7d289 refactor: migrate consumers off rustfs-common heal/scanner shims (#6623)
* refactor(ecstore): import heal/scanner contracts crates directly (backlog#1843)

* refactor(heal): import heal/scanner contracts crates directly (backlog#1843)

* refactor(lifecycle): import heal/scanner contracts crates directly (backlog#1843)

* refactor(obs): import heal/scanner contracts crates directly (backlog#1843)

* refactor(protos): import heal/scanner contracts crates directly (backlog#1843)

* refactor(scanner): import heal/scanner contracts crates directly (backlog#1843)

* refactor(rustfs): import heal/scanner contracts crates directly (backlog#1843)
2026-08-26 21:13:18 +08:00
Zhengchao AnandGitHub 2ada8a5cfb test(ci): stabilize main verification gates (#6661) 2026-08-26 18:57:40 +08:00
Zhengchao AnandGitHub 6ef7bac071 test(connect): isolate offline enrollment e2e root (#6659)
* test(connect): isolate offline enrollment e2e root

* test(connect): own e2e issuer key id
2026-08-26 18:50:58 +08:00
Zhengchao AnandGitHub 9ea5cd59ca fix(connect): ignore directory link-count churn (#6658) 2026-08-26 18:50:41 +08:00
Zhengchao AnandGitHub ff47714363 fix(tests): stabilize main CI concurrency fixtures (#6655)
* fix(tests): retry transient inventory replacement failures

* test(rio): isolate h2 keepalive fixture runtime
2026-08-26 18:16:47 +08:00
Zhengchao AnandGitHub 2ef9519bea fix(tests): tolerate sparse resume shard fixtures (#6653) 2026-08-26 16:47:18 +08:00
Zhengchao AnandGitHub b059569744 fix(ci): refresh Linux full E2E selection (#6649) 2026-08-26 16:10:19 +08:00
Zhengchao AnandGitHub 08d16d067d fix(ci): suppress macOS-only dead-code warning for INVENTORY_UID (#6641)
The INVENTORY_UID constant is only referenced inside
#[cfg(target_os = "linux")] test functions, so it appears unused on
macOS. Add a cfg_attr to allow dead_code on non-linux targets.
2026-08-26 15:11:16 +08:00
Zhengchao AnandGitHub 3c1e172600 fix(tests): stabilize Connect inventory fixtures (#6645)
* fix(tests): create inventory fixtures securely

* fix(tests): wait for inventory retry request
2026-08-26 15:08:59 +08:00
Zhengchao AnandGitHub 4cd38feae7 fix(tests): restrict Connect state tempdirs (#6642) 2026-08-26 14:29:51 +08:00
Zhengchao AnandGitHub 51041e917e fix(tests): repair main CI fixtures (#6636) 2026-08-26 13:57:37 +08:00
Zhengchao AnandGitHub 45f706b274 test(iam): pin the policy-to-iam error mapping and record the fold verdict (#6635)
Backlog#1845 step 8 conclusion. The plan called for folding iam::Error into a policy::Error #[from] wrapper and deleting the hand-written mapping. Measurement rejected the fold: the duplicated variants have ~220 construction/match sites (about 140 in production) across iam and the admin handlers - all auth-critical - and the alias route is blocked by the orphan rule (iam's From<IamStorageError> and io conversions cannot be implemented for a foreign type). Meanwhile the drift risk the fold targeted is already compiler-covered: the From match is exhaustive with no catch-all, so any new policy variant fails the build until mapped.

What remains of the step, delivered: the six dead policy variants are gone (previous commit), the grouped lossy arm is down to the two variants actually produced, a doc comment on the From impl records the verdict with the evidence, and a new totality test constructs one representative of every policy::error::Error variant and asserts the conversion preserves the rendered message - so the mapping is now pinned loss-free in both directions the classifiers care about.

Ref rustfs/backlog#1845
2026-08-26 13:30:23 +08:00
Zhengchao AnandGitHub 2b61990ec6 refactor(heal): classify recoverability typed-first with documented fallback (#6629)
refactor(heal): classify recoverability typed-first with documented needle fallback

Backlog#1845 step 6. Heal's retry decision leaned on substring matching of rendered messages; the typed information available in the error values now takes priority:

- Lock failures classify by LockError's own taxonomy instead of the blanket Lock(_) => recoverable: fatal variants (ResourceNotFound / PermissionDenied / Configuration) are terminal since retrying cannot fix them, while contention and transport variants (Timeout, Network, Internal, AlreadyLocked, QuorumNotReached, InsufficientNodes, ...) stay recoverable exactly as before.
- DiskError::RemoteClientUnavailable and its StorageError twin (typed in #6619) join the typed recoverable lists, so client-acquisition failures no longer depend on which needle happens to appear in the detail.
- task.rs is_transient_lock_or_timeout_error consults LockError::is_retryable / QuorumNotReached and the typed Timeout variants before falling back to needles.
- The substring list is demoted to a documented fallback: every needle now carries a producer census comment naming what emits it, with the shrink-only rule stated (delete the needle when its producer becomes typed end-to-end). heal rename incomplete remains the one needle with no typed producer.

heal gains a direct rustfs-lock dependency (already transitive via ecstore) to name LockError variants.

New tests pin each typed source: contention/transport lock variants recoverable, fatal lock variants terminal, RemoteClientUnavailable recoverable with a detail that avoids every needle. All existing recoverability tests stay green.

Ref rustfs/backlog#1845
2026-08-26 13:25:33 +08:00
Zhengchao AnandGitHub b610d5a55d fix(policy): remove six dead error variants (#6631)
Backlog#1845 step 8 prerequisite. policy::error::Error carried six variants with zero construction and zero match sites anywhere in the workspace: ErrCredMalformed, CredNotInitialized, NoAccessKey, InvalidToken, InvalidAccessKey, InvalidExpiration. Their only reference was the grouped fallthrough arm in iam's From<policy::error::Error>, whose own dead same-name twins were already removed in backlog#1831 (#6030).

Delete the variants and their display-message test rows; the iam mapping's grouped arm shrinks from eight variants to the two that are actually produced (InvalidServiceType from service_type parsing, JWTError via #[from]). This clears the way for folding the remaining 25-arm hand-written mapping (backlog#1845 step 8).

Ref rustfs/backlog#1845
2026-08-26 13:24:36 +08:00
Zhengchao AnandGitHub 8f0d4a20d1 refactor(ecstore): extract the embedded S3 client into rustfs-s3-client (#6627)
The storage engine embedded a ~8.4K-line hand-written S3 HTTP client under crates/ecstore/src/client (rustfs/backlog#1842). That client is a legitimate engine capability — it consumes remote S3-compatible endpoints for ILM tier warm backends and transition targets — but it was misfiled inside the engine, dragging s3s/hyper wire types into ecstore and blocking ARCHITECTURE.md invariant 4.

This PR is the pure-move step: 21 modules move verbatim to the new crates/s3-client crate (rustfs-s3-client), and crates/ecstore/src/client/mod.rs becomes a re-export shim so every in-crate crate::client:: path keeps working. The two server-side modules that were historically misfiled under client/ — object_api_utils.rs and object_handlers_common.rs — stay in ecstore.

Three reverse dependencies from the client into engine internals are severed so the move can be pure:

- transition_api::ReaderImpl::ObjectBody held ecstore's GetObjectReader; the client only ever reads the body, so the variant now holds an ObjectReader newtype over Box<dyn AsyncRead + Send + Sync + Unpin> with the same read_all() surface. The single production construction site (set_disk transition upload) and the two engine-side consumers were adjusted.
- api_list/api_remove used ecstore's storage_api_contracts / object_api types; api_list now imports BucketInfo from rustfs-storage-api directly, and api_remove uses the client's own transition_api::ObjectInfo (only .name/.version_id were read; the error-path bucket name is now threaded as a parameter instead of read from the deleted objects).
- the api_put_object_streaming regression tests built a GetObjectReader by hand; they now wrap the duplex stream in ObjectReader::new.

Guard updates: the s3s footprint ratchet gains an ecstore-scoped counter (42 files, shrink-only, per rustfs/backlog#1842), the ecstore module-lint-blanket register follows the moved files into crates/s3-client so the blanket ratchet keeps covering them, the logging guardrail path pin follows transition_api.rs, and the ::other(format!) baseline is regenerated (moved call sites left ecstore).

Verification: cargo check -p rustfs-s3-client -p rustfs-ecstore; cargo nextest run -p rustfs-s3-client (43 passed) and -p rustfs-ecstore (4515/4523; the 8 failures reproduce identically on pristine origin/main on the same machine); cargo clippy --all-targets; scripts/check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_s3s_footprint.sh, check_logging_guardrails.sh, check_error_other_format_ratchet.sh, check_doc_paths.sh, check_ci_paths_sync.sh all pass.
2026-08-26 12:38:52 +08:00
Zhengchao AnandGitHub 65a7cc9cd4 refactor(replication): name the resync state error and keep io failures typed (#6628)
Backlog#1845 step 7. The replication crate's hand-rolled, crate-generic Error type actually describes one thing: failures of the persisted resync/MRF state files. Rename it to ResyncStateError so the name says so, and stop collapsing io::Error into Other(String): a new Io(std::io::Error) variant keeps the kind and source chain, Display renders identically, and the ecstore boundary maps it to StorageError::Io so the kind survives into store-layer classification instead of degrading into a stringified other().

No thiserror introduced - the crate keeps its zero-internal-deps posture and hand-written impls.

Ref rustfs/backlog#1845
2026-08-26 12:32:53 +08:00
Zhengchao AnandGitHub aa56d4b847 refactor(ecstore): make store-to-disk error narrowing named and fallible (#6626)
refactor(ecstore): make store-to-disk error narrowing a named fallible operation

Backlog#1845 step 4. The blanket impl From<StorageError> for DiskError let ? silently push store-only errors (locks, buckets, quotas) across the disk boundary into DiskError::other, where the rendered message fragments reduce_errs quorum buckets. Same story for the blanket From<StorageError> for rustfs_filemeta::Error and its other() catch-all.

Both impls are replaced by named, fallible methods: StorageError::narrow_to_disk() and StorageError::narrow_to_filemeta(). Variants with an identity on the far side map across unchanged - including the two documented lossy collapses (SlowDown -> TooManyOpenFiles, StorageFull -> DiskFull) that the round-trip tests pin - and everything else returns Err(self) so the call site decides what crossing the boundary means. Removing the impls let the compiler enumerate every conversion site; the census that scoped this issue had found 5, the compiler found 33.

Call sites keep their existing behavior: the io identity bridge and the generic sites fold Err into the io-backed other() exactly as the old catch-all did (identity still recoverable by downcast), listing paths use one shared to_filemeta_err helper, and the two sites that relied on the SlowDown collapse now construct DiskError::TooManyOpenFiles directly so the loss is visible where it happens. No behavior change intended anywhere; the io::Error bridge itself is untouched by design.

Ref rustfs/backlog#1845
2026-08-26 12:32:37 +08:00
Zhengchao AnandGitHub a1ed41e109 fix(ci): refresh Linux e2e full selection (#6625) 2026-08-26 12:01:33 +08:00
Zhengchao AnandGitHub 4bc9dc482a fix(ci): release Connect test locks before await (#6620) 2026-08-26 11:37:07 +08:00
Zhengchao AnandGitHub c0c208d89a feat(ecstore): type internode client-acquisition failures for quorum buckets (#6619)
* feat(ecstore): type internode client-acquisition failures for stable quorum buckets

Backlog#1845 step 3, first typed family. The largest other(format!) message family in ecstore was 'can not get client, err: {detail}' (~50 production sites): every internode RPC that fails to acquire a client wrapped the dial/auth error with per-peer detail into DiskError::other / StorageError::other, whose Io equality compares the rendered message. N disks failing for this same cause therefore counted as N distinct errors in reduce_errs, starving quorum aggregation, and remote_disk call sites double-wrapped the message on top of get_client's own wrap.

Introduce DiskError::RemoteClientUnavailable(String) (wire code 0x2B) and its StorageError twin (StorageErrorCode 0x54): equality and hashing use the wire code alone, so same-cause failures land in one quorum bucket regardless of per-peer detail, while Display keeps the detail so substring classifiers (network needles, heal recoverability) keep reading it unchanged. Wire encoding carries the rendered detail in error_info and decode restores the typed variant; old peers fall back to the legacy string form gracefully.

Call sites: remote_disk get_client/get_bulk_client/offline-bypass/recovery-probe now construct the typed variant and the ~60 redundant double-wrap map_errs are gone; peer_rest_client's three client getters and offline gates, peer_s3_client, and admin_server_info follow. The tier-config-reload connection classifier's anchored 'can not get client' substring check becomes a typed match on the variant (the string form is retired and now classifies as Terminal, pinned by test).

Ref rustfs/backlog#1845

* chore(ci): refresh error other ratchet baseline

* fix(ecstore): classify typed client network failures
2026-08-26 11:24:56 +08:00
Zhengchao AnandGitHub 7cac528de3 refactor(protos): move compat manifest send-site assertions into owning crates (#6618)
refactor(protos): move internode compat manifest send-site assertions into owning crates

Promotes the rolling-upgrade dual-write manifest from a test-only constant in rustfs-protos into the public rustfs_protos::compat_manifest module, moves the JSON-encoder send-site assertions into the crates that own the asserted sources (ecstore remote_disk.rs for requests, the rustfs binary node_service/disk.rs for responses), and splits the scanner Phase-0 overlap inventory so its heal- and ecstore-owned halves live in those crates. Adds a cross-crate include_str!/include! guard with fixture self-tests to scripts/check_layer_dependencies.sh so a library crate can never again read another crate's Rust source at compile time, and records the rule in docs/architecture/crate-boundaries.md.

Part of rustfs/backlog#1884.
2026-08-26 11:11:16 +08:00
Zhengchao AnandGitHub eaf0d4da81 fix(ci): keep Connect integration fixtures public-only (#6617) 2026-08-26 11:09:51 +08:00
Zhengchao AnandGitHub bab9049cb0 fix(ci): clear remaining merged main Clippy lints (#6616) 2026-08-26 10:50:49 +08:00
Zhengchao AnandGitHub df5ae13fd0 refactor(common): move scanner/heal contracts into dedicated crates (#6615)
refactor(common): move scanner/heal domain contracts into dedicated crates

crates/common carried ~5.6K lines of scanner/heal domain code (metrics.rs,
heal_channel.rs, last_minute.rs) parked there to break dependency cycles;
every scanner type change recompiled all 11 rustfs-common dependents.

Pure move, zero renames, zero shape changes (backlog#1843):

- New crate rustfs-heal-contracts receives heal_channel.
- New crate rustfs-scanner-contracts receives metrics, last_minute, and the
  GLOBAL_INIT_TIME trio (metrics::report() reads it as the current-cycle
  fallback, so it must live below the shim to avoid a dependency cycle).
- rustfs-common re-exports everything at the old paths as a transitional
  shim; consumers migrate crate by crate, then the shims are deleted.
2026-08-26 10:49:12 +08:00
Zhengchao AnandGitHub 37a50f1e5d test(ecstore): pin error conversion round-trips for heal-matched variants (#6613)
Backlog#1845 step 1 (pure tests, no behavior change): pin the current behavior of every conversion seam an error crosses before heal, replication, or quorum aggregation classifies it, so the later typed-variant and narrow_to_disk() refactors change these expectations deliberately rather than silently.

Covered seams: DiskError <-> StorageError, DiskError <-> node_service wire Error, DiskError/StorageError <-> io::Error (the by-design identity bridge), and StorageError <-> rustfs_filemeta::Error.

Documented lossy edges pinned as-is: SlowDown collapses to TooManyOpenFiles across the disk boundary (StorageFull to DiskFull likewise), the wire Io catch-all re-wraps the rendered message on every hop and drops the io::ErrorKind, and other(format!) messages with per-disk detail fragment reduce_errs quorum buckets while identical messages still bucket together.

Ref rustfs/backlog#1845
2026-08-26 10:32:46 +08:00
Zhengchao AnandGitHub 4f4d268155 ci: ratchet ecstore ::other(format!) error construction shrink-only (#6614)
Backlog#1845 step 2. reduce_errs buckets per-disk errors by equality, and Io equality compares the rendered message, so an other(format!(..)) error embedding per-disk detail makes N same-cause failures count as N distinct errors during quorum aggregation. The census that opened the issue counted 1,609 such sites; the production count in crates/ecstore/src is 657 today and was still growing.

Freeze it: scripts/check_error_other_format_ratchet.sh counts ::other(format! sites per file (trailing #[cfg(test)] modules excluded) against a shrink-only per-file baseline, failing on any growth and on stale entries after a shrink, following the layer-dependency-baseline model. Wired into make pre-commit / pre-pr / dev-check and the CI Quick Checks job.

Ref rustfs/backlog#1845
2026-08-26 10:21:25 +08:00
Zhengchao AnandGitHub 9db1d6f06b fix(ci): restore merged main Clippy lanes (#6612) 2026-08-26 10:19:47 +08:00
Zhengchao AnandGitHub 45c03ca37f test(obs): move source-text logging tests into the logging guardrail script (#6610)
test(obs): replace source-text logging tests with logging guardrail script coverage

The seven fs::read_to_string source-text tests in crates/obs/src/logging.rs asserted retired logging patterns and required structured-logging fields across 13 files in other crates, four of them reverse reads into the rustfs binary crate. Their patterns are now enforced by scripts/check_logging_guardrails.sh, which runs in pre-commit and CI, covers the same files through checked_files plus require_patterns, and does not silently lapse when a governed file moves.

Part of rustfs/backlog#1884.
2026-08-26 09:55:49 +08:00
Zhengchao AnandGitHub 0ad6bf72cb fix(ci): restore merged main static gates (#6609) 2026-08-26 09:51:37 +08:00
Zhengchao AnandGitHub 5243bee746 ci: stop daily freshness false alarms for dormant scheduled workflows (#6608) 2026-08-26 09:51:22 +08:00
Zhengchao AnandGitHub 42d47b5f1e docs(release): require confirmation after preview validation (#6607) 2026-08-26 09:51:18 +08:00
Zhengchao AnandGitHub ba629bdae0 feat(connect): rotate device credentials at runtime (#6586)
* feat(connect): rotate credentials from heartbeat runtime

* fix(connect): make rotation safe with in-flight telemetry

* fix(connect): keep rotation retry state private

* fix(connect): preserve public rotation retries

* fix(connect): preserve heartbeat error API

* fix(connect): keep heartbeat alive during reenrollment

* fix(connect): validate pending reenrollment before skipping

* fix(connect): validate pending reenrollment token

* fix(connect): bind pending reenrollment state

* fix(connect): recover credentials before telemetry
2026-08-26 09:34:57 +08:00
Zhengchao AnandGitHub 5cce18fef2 test(diagnose): run binary smoke in CI (#6605) 2026-08-26 09:34:21 +08:00
Zhengchao AnandGitHub 54e2dce495 ci: run live target backend tests (#6603)
* ci: run live target backend tests

* test(targets): align MySQL live assertions
2026-08-26 09:34:11 +08:00
Zhengchao AnandGitHub 6f0a371f01 test(e2e): require exact object lock rejection oracles (#6580) 2026-08-26 09:34:05 +08:00
Zhengchao AnandGitHub 1bcb396752 fix(ecstore): version pool metadata transactions (#6604)
* fix(connect): adapt offline array predicate

* test(e2e): update smoke selection baseline

* test(ecstore): make slowtail oracle deterministic

* test(get): stage relocated fixture after reader opens

* ci: bound feature test link concurrency

* test: give lifecycle transition futures a larger stack

* fix(ecstore): version pool metadata transactions
2026-08-26 09:33:51 +08:00
Zhengchao AnandGitHub c0c5fc22f9 ci: preserve ILM timeout diagnostics (#6602)
* fix(connect): adapt offline array predicate

* test(e2e): update smoke selection baseline

* test(ecstore): make slowtail oracle deterministic

* test(get): stage relocated fixture after reader opens

* ci: bound feature test link concurrency

* test: give lifecycle transition futures a larger stack

* ci: preserve ILM timeout diagnostics
2026-08-26 09:33:42 +08:00
Zhengchao AnandGitHub 6886f7cac4 test(ci): give lifecycle transitions a larger stack (#6595)
* fix(connect): adapt offline array predicate

* test(e2e): update smoke selection baseline

* test(ecstore): make slowtail oracle deterministic

* test(get): stage relocated fixture after reader opens

* ci: bound feature test link concurrency

* test: give lifecycle transition futures a larger stack
2026-08-26 09:33:25 +08:00
Zhengchao AnandGitHub 032c5f9ac6 ci: bound feature test link concurrency (#6594)
* fix(connect): adapt offline array predicate

* test(e2e): update smoke selection baseline

* test(ecstore): make slowtail oracle deterministic

* test(get): stage relocated fixture after reader opens

* ci: bound feature test link concurrency
2026-08-26 09:33:16 +08:00
Zhengchao AnandGitHub 4e749d7046 test(get): stage relocated fixture after reader opens (#6584)
* fix(connect): adapt offline array predicate

* test(e2e): update smoke selection baseline

* test(ecstore): make slowtail oracle deterministic

* test(get): stage relocated fixture after reader opens
2026-08-26 09:33:07 +08:00
Zhengchao AnandGitHub 51449f0975 test(e2e): update smoke selection baseline (#6582)
* fix(connect): adapt offline array predicate

* test(e2e): update smoke selection baseline

* test(ecstore): make slowtail oracle deterministic (#6583)
2026-08-26 09:32:53 +08:00
Zhengchao AnandGitHub 90f64c60af fix(connect): adapt offline array predicate (#6581) 2026-08-26 09:32:40 +08:00
Zhengchao AnandGitHub 2bd83a5276 feat(connect): build signed offline bundles (#6579)
* feat(connect): build signed offline bundles

* fix(connect): validate offline bundle inputs

* test(connect): use the target architecture

* chore(connect): scope the unsafe allowance
2026-08-25 21:37:44 +08:00
Zhengchao AnandGitHub 6a99edab50 fix(s3): reject tampered multipart payloads cleanly (#6578) 2026-08-25 21:37:28 +08:00
Zhengchao AnandGitHub be1562e089 test(heal): wait for versioned fixture copies (#6571) 2026-08-25 21:21:52 +08:00
Zhengchao AnandGitHub b4a78fc907 fix(api): preserve typed upload digest errors (#6564) 2026-08-25 21:20:13 +08:00
Zhengchao AnandGitHub 02317dd36f test(keycloak): fix Keycloak OIDC live fixture (#6563) 2026-08-25 21:20:03 +08:00