e388a3ff5380f67f201c587dd2b8d7037978e904
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 唐小鸭andGitHub
|
7c4e514ec9 | fix(sse): document and lock anonymous denial under KMS key policy (#6739) | ||
|
唐小鸭andGitHub
|
c95b4f0820 |
test(ecstore): make config tests robust to DEFAULT_KVS registration order (#6731)
Config::new() and the external decode path read the process-global DEFAULT_KVS OnceLock at call time, and config::tests in the same test binary register it via crate::config::init() mid-run. Several com.rs tests asserted on unregistered state (heal section absence, equality with a later Config::new()), so they could flip depending on thread scheduling under cargo test -p rustfs-ecstore --lib config::. Assert on the semantic heal diff instead of section presence, normalize compared configs with a single DEFAULT_KVS snapshot taken after both sides exist, and compare the snapshot transaction test against the persisted baseline bytes. |
||
|
唐小鸭andGitHub
|
daeaf40e2c |
test: deflake config snapshot, presigned tamper, and pool resume tests (#6721)
* test(ecstore): decouple server config snapshot test from global defaults The final assertion of server_config_snapshot_serializes_read_modify_write_transactions compared the second snapshot against a fresh Config::new(). Config::new() reads the process-global DEFAULT_KVS OnceLock, which a sibling test in the same process can register mid-run (crate::config::init()), so the in-process run 'cargo test -p rustfs-ecstore --lib config::' failed while nextest's process-per-test isolation hid the coupling. Assert on the snapshot's raw bytes against the baseline blob instead, which is deterministic and matches the invariant under test: the second transaction observes the store unchanged by the first. * test: deflake presigned tamper helper and relocated-pool resume staging tamper_signature only remapped '0' and 'a', so a signature containing neither (about 1 in 5000) left the URI unchanged and tripped the helper's own guard assert in CI. Complement every hex digit (15 - v) instead: the map has no fixed point, so the tamper always changes the value while keeping length and hex shape. execute_get_object_resumes_from_relocated_pool_without_splicing_body staged the relocation by reading xl.meta from every source-pool disk, but a write-quorum commit legitimately leaves a lagging minority disk without the object directory (#6701) — the test already tolerates that gap when normalizing the upload pool, and CI suite IO load hit the same gap in the staging loop. Skip sourceless disks, carry the staged metadata path explicitly, and assert a write-quorum majority was staged. |
||
|
唐小鸭andGitHub
|
b90443f697 | fix(config): tolerate legacy scalar heal/scanner config sections (#6691) | ||
|
唐小鸭andGitHub
|
286626c1bd | feat(kms): bulk DEK rekey sweep with admin API and kms:Rekey action (#6654) | ||
|
|
7ecb44ea60 |
feat(kms): object-level DEK rewrap adapter and Transit context-bound rewrap (#6644)
* feat(kms): object-level DEK rewrap adapter and Transit context-bound rewrap * fix(kms): zeroize rewrap plaintext on cancellation --------- Co-authored-by: overtrue <[email protected]> |
||
|
唐小鸭andGitHub
|
a4377b6351 | feat(kms): bind encryption context into DEK envelopes as AAD (#6639) | ||
|
唐小鸭andGitHub
|
32346f159a | feat(kms): wire Vault custom CA and mTLS client identity (#6638) | ||
|
唐小鸭andGitHub
|
b1b3655bf8 | feat(kms): surface non-production backend positioning at runtime (#6633) | ||
|
唐小鸭andGitHub
|
9118a6e344 |
feat(ecstore): closed-form range seek for single-part v2 encrypted objects (#6601)
Single-part encrypted objects in the legacy format could not serve range reads without decrypting from byte 0: v1 frames are emitted per upstream read, so no closed-form plaintext-to-ciphertext mapping exists. The v2 layout fixed the frame length (8218 ciphertext bytes per 8 KiB plaintext frame), making the mapping closed-form. Consume it: - Single-part PUTs that encrypt locally under the v2 write switch stamp the frame-layout marker, valued with the object's data_dir token - ciphertext passthrough, data movement and copies mint a new data_dir or strip the marker, so a re-homed marker disqualifies itself. - The encrypted read plan seeks marked, uncompressed single-part objects to frame_index * 8218 and decrypts from that frame: the frame index rides the plan's sequence-number slot into DecryptReader::new_at_block, whose nonce and AAD bind absolute indices. New metric path label frame_seek. - A lying marker fails closed: v2 authentication rejects bytes at a fake frame boundary; plaintext is never served from the wrong offset. Compressed objects and multipart sub-part seeks keep the conservative paths (follow-up work); reading needs no switch - seekability follows the marker. |
||
|
唐小鸭andGitHub
|
f469869620 |
feat(rio): authenticated fixed-frame v2 encryption layout behind a write switch (#6600)
The legacy rio v1 stream format authenticates only each frame's ciphertext: the 8-byte header (length + plaintext CRC32) and the end marker sit outside the AEAD, frames carry no position binding, and nothing marks the last frame - header rewrites, frame reordering and truncation of trailing frames are not cryptographically detected. Add a v2 layout in the same format family, dispatched per frame by the type byte: - the header plus the frame's index are AEAD associated data (0x01), so header tampering, reordering and cross-position splicing fail authentication; - the final frame carries its own authenticated type byte (0x02); a clean EOF or an end marker before it is an error, every stream (including the empty one) ends in an authenticated final frame, and a v2 multipart stream fails if it ends before all listed part segments; - the writer accumulates full 8 KiB blocks, so non-final frames are fixed-length (8218 ciphertext bytes) and single-part objects gain a closed-form offset mapping for the follow-up range seek. Key hierarchy, nonce derivation, envelopes and metadata are unchanged; v1 objects stay readable forever, while v2 frames reject the historical nonce fallbacks and unknown frame types become a hard error. Write side ships off by default (RUSTFS_ENCRYPTION_FRAME_V2): mixed version clusters cannot read v2 frames, and encrypted ciphertext travels verbatim through transition, decommission and SSE-C replication passthrough. This release ships read support; the default flips in a following release. |
||
|
唐小鸭andGitHub
|
5834949c56 |
feat(kms): seal persisted config secrets with RUSTFS_KMS_CONFIG_SECRET (#6599)
The dynamic-configuration flow persisted KmsConfig to cluster storage as raw JSON, leaving inline authentication material - the Vault token, an AppRole secret_id, the Local master key - in config/kms_config.json in cleartext. Add rustfs_kms::config_secret: with the per-node RUSTFS_KMS_CONFIG_SECRET set, those field values are sealed in place before persistence (Argon2id with the Local key store's parameters + AES-256-GCM, per-value random salt, the field's logical label bound as AEAD associated data so sealed values cannot be swapped between fields). Sealed values carry the versioned prefix RUSTFS-KMS-ENC[v1]:. Compatibility is warn-only by owner decision: an unset secret keeps the plaintext format and warns naming the exposed fields; plaintext values load forever and reseal on the next save. Sealed values fail closed on a missing or wrong secret. The sealing secret must be an independent trust root - reusing the Local master key or Static secret is refused, mirroring the backup-KEK rule. |
||
|
唐小鸭andGitHub
|
f51b06f0ae |
perf(ecstore): enable encrypted range part-seek by default (#6598)
Range GETs on encrypted objects read the whole ciphertext from offset 0 and discarded the decrypted prefix, because the part-boundary seek shipped behind RUSTFS_ENCRYPTED_RANGE_SEEK defaulted to false (backlog#1316 Phase A). Flip the default to true. Safety rests on the marker chain: MPUs created without a candidate layout marker never become seek-eligible, CompleteMultipartUpload promotes the candidate to the quorum marker only after revalidating it against the object's data_dir under the uploadId write lock, and reads seek only when the quorum marker matches the current data_dir. Single-part, compressed and markerless objects keep the full-read path; RUSTFS_ENCRYPTED_RANGE_SEEK=false remains the kill switch. The stale default-off regression test becomes test_legacy_range_seek_defaults_enabled: the unset-env default must match the explicit opt-in plan, seek past the leading parts, and not span the whole ciphertext. |
||
|
唐小鸭andGitHub
|
a65b306fb0 |
perf(sse): drop the second KMS decrypt from encrypted GET responses (#6597)
perf(sse): classify GET response headers without a second KMS unwrap An SSE-KMS GET performed two backend Decrypt calls per request: the object layer's encryption resolver unwraps the envelope to build the decrypted stream, and the S3 layer then called sse_decryption again purely to derive response headers, discarding the returned key bytes. Replace the S3-layer call with classify_sse_read_response, which reproduces that call's behavior from stored metadata alone: SSE-C validation errors and precedence, per-key kms:Decrypt authorization ahead of every other failure mode, and the request's KMS audit summary fields. The success outcome stays honest because a failed unwrap aborts the read in the object layer before response classification is reached. Tests cover header parity against the unwrap-based path, audit-tag parity for allowed and denied principals, SSE-C validation parity, and prove classification needs no DEK provider at all. |
||
|
唐小鸭andGitHub
|
3d75e7b51f |
fix(ecstore): heap-allocate durable ILM receipt futures (#6527)
PR #6369 awaits record_durable_ilm_decommission_progress/terminal inline from save/delete_transition_transaction_record. Their state machines are large and sit on the already-deep transition worker poll chain (worker -> transition -> transaction record -> delete_config -> full store delete fanout), which overflowed the default 2 MiB tokio worker stack in debug builds: app::lifecycle_transition_api_test:: compensation_driven_complete_multipart_upload_still_transitions died with SIGABRT in under a second (first-bad commit via git bisect 1.0.0-rc.3..1ec1a8d90: |
||
|
|
fc98dbb654 |
fix(replication): tolerate orphaned resync intents at startup (#6470)
* fix(replication): tolerate orphaned resync intents at startup Since #5215 (1.0.0-beta.12) startup reconciles every pending/started resync intent in resync.bin against the bucket's configured targets and aborts the whole server when an intent has no matching target ARN. A resync whose remote target was later removed leaves exactly such an orphan on disk, so every later start fails with "accepted replication resync target ... is not configured" regardless of the binary version. Skip orphaned intents with a warning instead of failing startup; the resync routine already settles them to ResyncFailed. Cancel the intent when its remote target is removed so the orphan is not created again. Fixes #4784 * fix(replication): cancel removed-target resync under the admission lock Canceling through this node's cached whole-bucket status map could persist a map that predates another node's admission, erasing that node's durable restart intent. Reload resync.bin under the bucket admission lock, publish the fresh map, and only then mark the removed target's intent canceled. Two-node regression covers the clobber. * fix(replication): persist resync status via ETag CAS merge mark_status, the periodic saver, admission, and removed-target cancellation all persisted their node's cached whole-bucket map, so any one node's stale cache could resurrect states another node had already finalized (a canceled intent flipping back to Pending, an admission vanishing). All resync.bin writers now go through update_resync_status_cas: load the freshest document with its ETag, apply a per-target mutation with staleness and canceled-is-terminal guards re-checked against the persisted entry, and save conditionally, retrying on concurrent writes. The periodic saver merges per target, letting terminal states and newer admissions recorded elsewhere win. Cache convergence stays per-target so locally running resyncs keep their authoritative progress counters. Regressions: stale_peer_status_write_cannot_resurrect_canceled_intent (node B's pre-cancel cache marking its own run Started must not revive node A's canceled intent) plus unit coverage for the periodic-save merge. * test(ecstore): rename resync test helper off the guarded contract name fn resync_target is on the architecture guard's reserved list for crates/replication operation contracts; the merge-test helper now reads resync_target_state. * fix(replication): serialize resync status updates --------- Co-authored-by: overtrue <[email protected]> |
||
|
唐小鸭andGitHub
|
3ce01dcc73 | fix(object-lock): unblock authorized replication writes on locked versions and tolerate cleared lock metadata (#6413) | ||
|
唐小鸭andGitHub
|
31933c32f9 | fix(replication): apply receiver-side LWW to inbound metadata categories (#6379) | ||
|
唐小鸭andGitHub
|
0e88a27d05 | fix(admin): use madmin key names in list-remote-targets response (#6377) | ||
|
唐小鸭andGitHub
|
35e264a9f5 | fix(admin): advertise IAM admin capabilities in runtime capabilities (#6336) | ||
|
|
450ec7f66a |
fix(admin): bound site replication lifecycle lock and parallelize add preflight (#6378)
The site replication add preflight probed peer sites serially while holding the process-wide lifecycle lock, so k unreachable sites held the lock for k peer-request timeouts, and every concurrent add/remove/refresh waited on an unbounded lock acquire for the whole time. Probe all sites concurrently (matching the file's other peer fan-outs) so k unreachable sites cost roughly one timeout, and bound the lifecycle lock acquire at 30s, returning a retryable 503 to waiters instead of hanging indefinitely. Regression tests pin the preflight fan-out concurrency, the bounded acquire's 503, and the 10s/3s peer client timeout constants. Refs rustfs/backlog#1952, rustfs/backlog#1946, rustfs/backlog#1889 Co-authored-by: houseme <[email protected]> |
||
|
|
4ddc728c9d |
fix(replication): deny non-owner replication config edits under site replication (#6375)
* fix(replication): deny non-owner replication config edits under site replication Under site replication a user holding only bucket-scoped s3:PutReplicationConfiguration could rewrite or erase the operator-managed site-repl-* rules, with the change broadcast to every peer (backlog#1948, audit A1/P2-17). - Gate PutBucketReplication/DeleteBucketReplication in the S3 handlers: when site replication is enabled and the requester is not the owner, return MinIO-parity XMinioReplicationDenyEdit (HTTP 400). The gate runs after policy authorization and only on the external S3 path; the reconciler and peer bucket-meta ingestion are unaffected. - Defense in depth in the bucket usecase: PUT merges the incoming config with the stored site-repl-* rules (same merge as peer ingestion) instead of overwriting verbatim; DELETE keeps the site-repl-* rules and never garbage-collects a bucket target a surviving site-replication rule still references. - Move is_site_replication_rule / merge_incoming_replication_config / replication_target_arn_deployment_id from the admin site-replication handler down to rustfs-replication so the app layer can reuse them without new layering violations. * fix(replication): scope site-owned rule detection to reconciler-derived rules The `site-repl-*` prefix alone classified any rule as site-owned, so on a bucket outside site replication an owner's `site-repl-user` rule survived DeleteBucketReplication (rule and target kept, success returned). Rule ids do not reserve that namespace. A rule is reconciler-owned only when it matches what the reconciler derives: id `site-repl-<deployment id>` for a current remote site replication peer and a destination ARN naming that same deployment id. The S3 put/delete path reads the remote peer set (empty when site replication is disabled) and keeps exactly those rules; everything else is operator state the request replaces or deletes. An incoming rule that claims a current peer's id is dropped so the reconciler rule's id stays unique. The peer ingestion path and the reconciler keep their prefix predicate unchanged. * fix(replication): keep operator rule priorities across site rule merges Merging stored site-replication rules into a PutBucketReplication body renumbered every rule 1..n in list order, rewriting the submitted policy: overlapping same-target rules submitted as priority 5 then 1 became 1 then 2, so the delete-marker-disabled rule won the replication decision. The reconciler and the peer-removal prune renumbered the same way. Operator priorities now stay verbatim everywhere; only the reconciler's derived rules move, to the lowest priorities no operator rule uses, via one pure helper shared by the S3 edit merge, the peer ingestion merge, the reconciler pass and the prune. Being a pure function of the rule list it is idempotent, so the reconciler's no-op check still holds after a merged write, and an on-disk config in the historical layout (operator rules 1..k, site rules k+1..n) yields the same bytes, so nothing is rewritten on upgrade. * fix(replication): pass site peer ids into the bucket usecase from the interface layer The review fix made the bucket usecase read the site-replication peer set through the admin handlers, an app->interface import the layer guard rejects. The S3 handlers (interface) now read the peer set and pass it in, so the usecase stays a pure function of its inputs; a state-read failure still fails the edit closed, just one layer up. * fix(replication): classify peer-ingested rules by the derived id/ARN contract The peer ingestion merge still treated every incoming `site-repl-*` id as reconciler-owned, so an owner-authored `site-repl-user` rule that the S3 merge now keeps on the editing site was dropped on every peer and the sites persisted different operator configs. The ingestion merge now classifies by the same derived contract as the S3 merge: a rule is the reconciler's only when its `site-repl-<id>` names the deployment its destination ARN targets and that deployment is a site of the cluster (the receiver's own id included, since the sender's rule towards the receiver names it). The reconciler, the peer-removal prune and the target-online probe switch from the id prefix to the derived shape as well, so the rule survives their passes too; rules in the derived shape that name a removed peer or this site are still rebuilt away. Regression: a PutBucketReplication merged on site A and ingested on site B keeps `site-repl-user` on both and the operator rule sets agree. * fix(replication): keep an operator role target through site rule merges The S3 and peer-ingestion merges cleared `Role` whenever it parsed as a site-replication ARN, which an owner-submitted remote target with an empty region (`arn:minio:replication::<id>:<bucket>`) also does. The merged config then selected the rule destination ARNs instead of the validated role target. Only a role naming a current site of the cluster is the holder's identity (the reconciler's per-peer target lookup reads it); every other role passed target validation and stays. The reconciler's repair pass applies the same rule. Regression: an owner role target survives both merges and `filter_target_arns` / `replication_target_arns` select it; a role naming a current peer is still cleared. * fix(replication): gate operator priority preservation on a peer contract probe Keeping operator rule priorities verbatim is not rolling-upgrade safe: a peer still running the pre-contract code renumbers every rule 1..n in list order on ingest and on each reconciler pass, so an upgraded site broadcasting `5,1` leaves that peer on `1,2` — which can select the other overlapping rule — and the sites never reconverge. Operator rules now merge under an explicit contract: - `OperatorRuleContract::Derived`: site rules are the derived id/ARN shape, operator priorities stay verbatim (the behavior of the previous commits). - `OperatorRuleContract::Legacy`: byte-for-byte what a pre-contract peer does — `site-repl-*` ids are all site rules, a site-replication-shaped `Role` is dropped, every rule is renumbered 1..n in list order. The S3 merge additionally lists the operator rules in priority order first, so the renumbering keeps their relative order and the winning rule per target is the one the operator submitted. The S3 PutBucketReplication/DeleteBucketReplication path probes every remote peer through the existing `peer/edit-capabilities` endpoint (capability `derived-rule-contract`; pre-contract peers answer `success:false` or 404) and merges under Derived only when every peer supports it; any refusal or probe failure pins that edit to Legacy. Every bucket-meta item this site sends (S3 hooks, bootstrap plan, retry snapshots, tombstones) carries `derivedRuleContract: true`; a receiver merges a payload without the marker the Legacy way, so an item from a pre-contract sender is handled exactly as its own peers handle it. Rolling upgrade: while any site runs the older code every edit is canonicalized cluster-wide (numbers lost, order kept); once the last site is upgraded the next edit keeps its priorities. Configs canonicalized during the mixed period are not renumbered back — the derived priority assignment is a no-op on the canonical layout — so an operator who wants the original values re-submits the config after the upgrade completes. Adding a site that runs the older code after priorities were preserved is not gated and would desynchronize that bucket until the next edit. --------- Co-authored-by: houseme <[email protected]> |
||
|
唐小鸭andGitHub
|
5f72209446 |
fix(ecstore): keep unknown-size sentinel in create_bitrot_writer (#6380)
SSE and compression wrap the payload so its length is unknown and advertise HashReader::SIZE_PRESERVE_LAYER (-1). Every layer preserved that sentinel except create_bitrot_writer, which clamped it to 0 before calling DiskAPI::create_file. RemoteDisk forwards that size verbatim in the put_file_stream query, so remote peers were told the body was empty. Since the authenticated put-file trailer (#5868) the receiver used the declared size to split body from trailer, turning the clamp into a fatal "auth trailer has trailing data" failure for every SSE PUT on multi-node deployments (rc.2). #6320 relaxed the receiver to only trust size > 0; this change fixes the sender so the sentinel survives end to end and the wire no longer conflates empty objects with unknown-length streams. Refs #6331 |
||
|
唐小鸭andGitHub
|
f7003dfddd |
fix(admin): four site-replication interop correctness fixes (B5-rc T2) (#6399)
* fix(admin): send versioningEnabled on site replication make-bucket ops The outbound make-with-versioning bucket-op query only carried operation/createdAt/lockEnabled. MinIO's own create-bucket hook sends versioningEnabled=true on this op, so align the outbound query with MinIO's site-replication make-bucket wire contract. Route both outbound builders (bootstrap plan and create-bucket hook) through one shared builder that always appends versioningEnabled=true. RustFS's own inbound handler force-enables versioning either way, so RustFS-to-RustFS behavior is unchanged; the MinIO release verified against (RELEASE.2025-09-07) also force-enables versioning regardless of the flag, so this aligns the wire contract rather than changing observable behavior there. * fix(admin): propagate purge-deleted-bucket errors in site replication The purge-deleted-bucket branch of the peer bucket-ops handler dropped the delete_bucket error and answered 200, so a peer-driven purge that failed (disk full, quorum loss) was reported as success while the bucket survived on this site. Tolerate only bucket-not-found (the purge raced an earlier replay or a local delete) and propagate every other error through ApiError like the sibling delete branches do. * fix(admin): derive fallback site deployment ID with UUIDv5 deployment_id_for_endpoint used DefaultHasher, whose algorithm is not guaranteed stable across Rust releases. The fallback fires when a peer response carries an empty deploymentID; the result is persisted in site-replication state, used for collision disambiguation, and broadcast to peers, so a toolchain bump could re-derive a different ID for the same endpoint. Note that the add preflight currently rejects that case upstream of this fallback. Derive UUIDv5 (NAMESPACE_URL) over the canonical endpoint instead, and log a structured warn when a peer metainfo response arrives without a deploymentID. Already persisted fallback IDs are non-empty and therefore never re-derived, so existing state is unaffected. * fix(admin): stream site replication devnull body without 1MB cap The site-replication devnull endpoint buffered the request body through read_plain_admin_body, which enforces the 1MB admin body cap. MinIO peers stream multi-megabyte probe bodies to this endpoint during site netperf link checks and expect an unbounded discard, so any larger probe got a 400 and was misreported as a broken link. Stream and discard the body chunk by chunk with no size cap instead, mirroring MinIO's io.Discard drain. The response stays 204 with an empty body. |
||
|
唐小鸭andGitHub
|
a930152d5a |
fix(admin): expose per-target disableProxy through remote target admin API (#6376)
The read-proxy selector already honors a target's disable_proxy flag (PR #6172), but the admin API still rejected the field, so the only way to set it was importing a MinIO-written bucket-targets.json. - move disableProxy from REMOTE_TARGET_UNSUPPORTED_FIELDS to REMOTE_TARGET_WRITABLE_FIELDS (set-remote-target create accepts it) - add TargetUpdateOp::Proxy so set-remote-target?update=true&proxy=true overlays only the proxy group (MinIO TargetUpdateType parity) - bump REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION 1 -> 2 and update the runtime capability pin tests - keep edge/edgeSyncBeforeExpiry rejected (no implementation behind them) - pin that a published TargetClient carries disable_proxy, the field the proxy-target selector consults Refs rustfs/backlog#1950 |
||
|
唐小鸭andGitHub
|
22b4ef9f0c | fix(get): name the failing object on mid-stream GET body failures (#6284) | ||
|
唐小鸭andGitHub
|
b8686b471a | fix(site-replication): clear pending_remove on join, surface failures (#6244) | ||
|
唐小鸭andGitHub
|
1d056d7605 |
test(e2e): pin bounded physical reads for compressed multipart range GETs (#6167)
Byte-exactness tests stay green if the compressed range seek regresses into decoding from byte zero: the returned bytes are still correct and only the read amplification explodes. Assert the cost side as well. The observation reuses rustfs_io_get_object_shard_read_observed_bytes_total, already emitted per shard read by the erasure layer, so no production code is instrumented. The OTLP collector learns to accumulate a second counter, keyed by its path/role/outcome labels rather than by data-point position, which is not stable across exports. Two failure modes the assertions guard against: - With RUSTFS_OBS_METER_INTERVAL=1, treating one unchanged sample as settled measures a delta of zero, because the range read's counter has not been exported yet. Settling now requires several consecutive equal samples. - An upper bound alone passes vacuously on a zero delta, so a lower bound turns "measured nothing" into a failure instead of a green run. Refs rustfs/rustfs#5957, backlog#1848. |
||
|
唐小鸭andGitHub
|
1cf0f7af15 |
feat(replication): split oversized hot-path functions, proxy unreplicated reads, and fail SSE-C passthrough closed (#6170)
* refactor(replication): split four oversized hot-path functions into focused helpers Pure-move decomposition of the four oversized functions flagged by the replication compatibility review (P1-18), unblocking migration milestone M2 which requires resyncer moves to stay mechanical: - resync_bucket (522 lines -> 61-line step sequence): leader lock, target resolution, walk/collector/worker spawning, and dispatch loop extracted into focused helpers; pure decision helpers (DTO builders, HEAD-result classification) separated from IO orchestration. - replicate_all (411 lines -> 113-line main body): initial target-info seeding, read/stat option builders, skip-path notes, target HEAD action resolution, and the multipart/single-put payload transport extracted as private free functions. - start_mrf_processor (306 lines -> 46-line spawn body): recovery guard, ledger load, per-entry replay (delete/object/metadata), and retained entry resolution extracted; retry bookkeeping semantics preserved exactly (inner continue-paths push inside helpers, outer Missed push stays in the loop). - apply_iam_item (255 lines -> match dispatch skeleton): one helper per IAM item type. No behavior change: log texts, error paths, event emissions, and metric counts are byte-identical; existing tests unchanged and green (238 ecstore replication/mrf/resync + 232 rustfs site-replication). * feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172) * feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets Implements the MinIO active-active read-proxy protocol (P1-5 of the replication compatibility review): when a GET/HEAD/GetObjectTagging/ PutObjectTagging/DeleteObjectTagging request fails locally with not-found and the bucket has replication targets, the request is proxied to the targets in rule order, mirroring bucket-replication.go proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget. Protocol surface: - Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed into ObjectOptions (proxy_request + proxy_header_set, matching MinIO ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY value is never re-proxied. Outbound client proxy calls send the marker as "true"; replication worker convergence HEADs send it as "false" so a peer's proxy layer cannot answer a convergence check by proxying back to the source (which would fake Completed without a PUT). - Target selection: new replication_proxy.rs get_proxy_targets — empty when the marker is set, versioning is suspended, or no replication config; otherwise filter_target_arns -> TargetClient lookup, skipping targets with proxying disabled. - TargetClient gains head_object_for_proxy/get_object (streaming) and the three tagging calls. Proxy calls never send the replication-check SSE-C exemption header; customer SSE-C keys are forwarded verbatim so the target performs real decryption. Conditional (If-*) headers are not forwarded (MinIO parity); Range and part_number are, with parts_count/tag_count/storage_class/expiration passed through. - Metrics: proxy counters now count only real client proxy traffic, MinIO-aligned (one total per proxied request, one failed when no target served it). The previous misattributed counters — replication worker HEAD/PUT (#2672) and local tagging operations (#2682) — are removed; ReplProxyMetric now maps the tagging counters instead of dropping them. e2e (fake_s3_target extended with tagging + header journaling): proxied GET body + outbound header contract (marker present, no replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero outbound requests, GetObjectTagging, and metric mapping unit tests. Rolling note: proxying only activates for buckets with replication targets; requests carrying the marker keep pre-upgrade behavior. Refs rustfs/backlog#1675 (P1-5) * fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178) SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport headers. A MinIO/generic-S3 target silently discards them, storing bare ciphertext with no decryption material — yet the PUT succeeded, so the object reported COMPLETED with a silently unreadable replica (backlog#1675 N2). Fail-closed design: - SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in BucketTargetSys per target ARN with a recording timestamp. Entries reset whenever the target is rebuilt, edited, or removed (arn_remotes_map lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes): an expired verdict in either direction is re-earned through the audit, so an Unsupported target recovers automatically after an upgrade (at most one wasted PUT+HEAD audit per bad target per TTL window) and a Supported verdict cannot outlive a backend swapped behind the same endpoint. - Replication worker (replicate_object and replicate_all): fresh Unsupported targets never receive the PUT — the attempt fails immediately into the normal MRF retry channel with a "run ?replication-check to re-probe" hint. Unknown or expired verdicts are audited: after the PUT the worker HEADs the replica back through the replication-check channel (source version id mapped through resolve_read_api_version_id, so null-version objects audit correctly) and requires SSE-C evidence (the echoed customer-algorithm header); missing evidence records Unsupported and fails the attempt. Convergence HEADs are audited the same way, so a broken ciphertext replica from an earlier attempt can never launder itself into COMPLETED via an ETag match. The gate/evidence policy is pure (replication_target_boundary, staleness folded in as an input) for the M2 worker migration. - replication-check grows an SsecPassthrough probe phase: a probe PUT carrying the live transport-header shape, HEAD-back for evidence, and a machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure. The probe verdict is synced into the runtime capability cache. Unlike VersionFidelity, a failed SsecPassthrough phase does NOT fail the target overall — it is a capability limit, not a broken replication contract, and a plaintext-only deployment against such a target must not turn red. - fake_s3_target: default mode now models a RustFS target (stores the transport headers, echoes SSE-C evidence); the new drop_unlisted_replication_headers mode models MinIO. The journal records whether a request carried transport headers. Receiver-echo verification: the replication-check HEAD exemption only skips SSE-C key validation; the response has always built sse-customer-algorithm from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver change was needed — pinned end to end by the replication-check e2e against a real RustFS target. Rolling-upgrade constraint: RustFS targets older than the replication-check HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail it outright), so SSE-C replication to such targets reports FAILED. This is deliberate — FAILED-and-retryable beats a silently undecryptable replica — and self-heals: once the target is upgraded, the next TTL expiry (or a manual ?replication-check re-probe) re-audits and records Supported. Plaintext and managed-SSE replication are unaffected. The capability cache is per-node; each node audits independently. Known limitations: - The audit judges evidence from the echoed customer-algorithm header only. A hypothetical target that preserves that one header while dropping other transport headers (partial-drop) would pass the audit; no known target behaves this way — observed targets drop the whole unknown-header family. - A mixed-version target cluster can flap the verdict between audits routed to different target nodes until the rollout completes; the TTL bounds how long each stale verdict persists. New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a header-dropping fake (FAILED + no second PUT via the capability cache, journal-asserted; red run showed the old COMPLETED), replication-check reports the SsecPassthrough phase Code while the target stays OK overall, SSE-C heal convergence after a real target outage, and SSE-C existing-object resync landing a REPLICA readable with the customer key. TTL expiry in both directions is pinned at the cache and gate seams. * refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) (#6180) * refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) Pure-move milestone M2 of the ECStore replication split (backlog#1675 P1-17): relocate the resyncer's IO-free decision helpers, with their unit tests, into the crates they already belong to by type ownership. No behavior change. Moved into crates/replication: - resync.rs: resync_status_duration - delete.rs: resync_existing_delete_replication_info, replicate_delete_outcome, target_delete_version_id, delete_marker_purge_version_id, delete_marker_purge_mrf_entry - object.rs: version_identity_drifted, is_replication_target_offline_error, SsecPassthroughCapability, SsecPassthroughGate, ssec_passthrough_gate, ssec_passthrough_evidence_present (param-demoted to the echoed customer-algorithm string; ECStore keeps the HeadObjectOutput adapter) - filemeta.rs: NULL_VERSION_ID wire literal (crate-owned copy per the filemeta-independence contract) ECStore rewiring (Rule #14: imports stay in *_boundary.rs): - resync/object-decision/target boundaries re-export the moved symbols; resyncer call sites are unchanged - bucket_target_sys keeps only the verdict cache + TTL and re-exports the capability enum so existing consumer paths keep compiling Not moved (signatures carry ECStore or aws-sdk types): verify_resync_head_result, resync_target_error_detail, the SdkError classifiers, the replicate_all_* option/info builders, and the env-coupled bounded_resync_max_jobs admission clamp. README milestone table updated. * chore(replication): retire the datatypes.rs relay early README sanctions retiring datatypes.rs ahead of M4. The module was a pure relay (resync boundary -> datatypes -> mod.rs facade) with no external consumer importing it directly, so the facade now re-exports ResyncStatusType from replication_resync_boundary and the relay file is deleted. Consumers stay behind the ECStore facade, keeping Migration Rule #15 intact — the original retirement wording ("consumers import through rustfs-replication directly") conflicted with that rule and is corrected in the README. * chore(arch): extend migration guards to the M2-moved decision contracts The adversarial review of the M2 move found the per-symbol ratchet in check_architecture_migration_rules.sh was not extended for the moved symbols, leaving them free to be redefined in ECStore or imported past their boundary without CI noticing: - resync definition pin + boundary fences gain resync_status_duration; - the object-decision boundary fences gain the five delete-family helpers (delete_marker_purge_mrf_entry, delete_marker_purge_version_id, replicate_delete_outcome, resync_existing_delete_replication_info, target_delete_version_id); - the target-boundary fence gains the SSE-C gate family, the offline classifier, and version_identity_drifted; - a new definition pin rejects ECStore redefinitions of the M2-moved fns/enums (ssec_passthrough_evidence_present deliberately excluded: ECStore keeps a thin HeadObjectOutput adapter under that name). Mutation-verified: a probe fn ssec_passthrough_gate under crates/ecstore/src/bucket/replication trips the new pin. Also anchors the intentionally-duplicated NULL_VERSION_ID wire literal from the filemeta side and tightens the M2 README note on bounded_resync_max_jobs. |
||
|
唐小鸭andGitHub
|
e0b87b0e7e | fix(site-replication): admit only verifiable peer-edit fences (#6123) | ||
|
唐小鸭andGitHub
|
d091554ffe | fix(kms): resolve Vault auth from the environment at startup and add Kubernetes auth (#6095) | ||
|
唐小鸭andGitHub
|
c04ee41cf0 | feat(site-replication): drain the retry queue from the reconcile tick (#6131) | ||
|
唐小鸭andGitHub
|
e26668e62c |
fix(ecstore): mint bucket-target ARNs in the madmin arn:minio partition (#6128)
* test(ecstore): pin madmin-compatible ARN partition contract
Red-light evidence for backlog#1675 P1-7: madmin-go's ParseARN
hard-rejects any ARN that does not start with 'arn:minio:', while RustFS
generates and only accepts 'arn:rustfs:'. mc/madmin tooling therefore
cannot decode RustFS remote-target listings, and MinIO-era replication
configs are rejected as StaleTarget when re-registered. The new tests
pin the target contract (generate arn:minio:, parse both partitions,
reject unknown partitions) and fail against the current single-partition
gate.
* fix(ecstore): mint bucket-target ARNs in the madmin arn:minio partition
madmin-go's ParseARN hard-rejects any partition other than 'arn:minio:',
so native mc/madmin tooling could not decode RustFS remote-target
listings, and re-registering a MinIO-era replication config failed its
StaleTarget check against freshly minted arn:rustfs: targets
(backlog#1675 P1-7, route A).
- ARN Display now emits 'arn:minio:'; FromStr accepts a {minio, rustfs}
partition whitelist (the legacy partition stays readable forever for
persisted bucket-targets.json / replication configs). The whitelist is
the only structural gate — BucketTargetType::from_str never fails —
so it deliberately rejects foreign partitions such as arn:aws:.
- No data migration: every runtime match between targets, rules and
stats keys is full-string equality, so existing arn:rustfs: targets
keep matching their persisted rules; site replication already
preserves MinIO-era ARNs on reconcile (pinned by existing tests).
- Rolling upgrade note: upgrade all cluster nodes before creating new
remote targets — a not-yet-upgraded node rejects remove-remote-target
for a freshly minted arn:minio: ARN with BucketRemoteArnInvalid.
- Out of scope: notification/SQS ARNs (crates/targets) keep the
arn:rustfs:sqs: partition; they have their own compatibility story.
|
||
|
|
dcf3e4b9e8 |
fix(replication): transport and persist LWW timestamps for tag, retention, and legal hold (#6129)
* test(replication): pin missing LWW timestamp header transport
Red-light tests for the replication timestamp three-header contract:
- put_object_headers_carry_replication_timestamp_headers pins that
PutObjectOptions::header() must emit the
x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-timestamp
headers when the internal timestamps are set (currently missing).
- test_put_opts_from_headers_gates_replication_timestamp_persistence_on_authorization
and test_complete_multipart_opts_persist_replication_timestamps_when_authorized
pin that an authorized replication PUT / multipart complete must persist
the inbound timestamps into the internal metadata keys while unauthorized
requests must not (currently never persisted).
- fake_s3_target journals the three timestamp headers per request
(ReplicationTimestampHeaders on RequestRecord) so sender-side e2e
assertions can observe what a real target receives; self-test included.
* fix(replication): transport and persist LWW timestamps for tag, retention, and legal hold
Active-active conflict resolution for concurrent tag/retention/legal-hold
edits needs the source's per-category modification times on both sides of
the wire; the three AdvancedPutOptions timestamp fields were dead and the
headers were neither sent nor parsed.
- Emit x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-
timestamp from PutObjectOptions::header(); names and RFC3339 values
interoperate with MinIO (minio-go constants.go, object-api-options.go),
pinned by a header_compat wire-name test.
- Default the three AdvancedPutOptions timestamps to UNIX_EPOCH and skip
epoch values in header(), so "never modified" is not sent as a
modification made now.
- Parse the headers only on authorized replication PUTs and multipart
completes, expose them as Option<OffsetDateTime> on ObjectOptions, and
persist them into the dual-prefix internal metadata keys so the
outbound pass (replication_target_boundary) reads the source's
timestamps instead of the mod_time fallback.
- Record the local tagging timestamp in the PutObjectTagging and
DeleteObjectTagging eval metadata, mirroring the object-lock handlers;
without it the sender only ever had the mod_time fallback to offer.
Receiver-side LWW comparison (keep newer stored category metadata over a
stale inbound copy) is left as a TODO at the parse site.
* fix(replication): load the stored tagging timestamp independently of remaining tags
Review: DeleteObjectTagging persists the tagging-timestamp internal key
but leaves the object tagless, and the outbound mapper only loaded the
key inside the user_tags-nonempty branch — the deletion's LWW timestamp
stayed at the epoch and the header was omitted, so the deletion could
never win conflict resolution on the replica. The stored key is now
loaded unconditionally; the mod_time fallback still applies only while
tags exist (MinIO parity), and a tagless object without the key keeps
the epoch default (no header). Deletion-path regression test added.
* fix(storage): reserve replication transport names at metadata ingest
Second review round: a client PUT of
x-amz-meta-x-rustfs-source-replication-tagging-timestamp materialized
the bare transport key as stored user metadata. The outbound
replication header builder forwards user metadata verbatim on a
server-authorized request, so the receiver would persist the
attacker-chosen value as trusted internal LWW state — and for a
tagless object nothing later overwrites it.
The ingest namespacing guard now reserves the whole
x-rustfs-source- / x-minio-source- families (the new timestamps and
their siblings: source-mtime/-etag/-version-id/-replication-request),
folding forged keys back under x-amz-meta-. Forged-ingress regression
covers both prefixes and a sibling.
* fix(replication): harden timestamp replay
* fix(app): route retention helper through facade
---------
Co-authored-by: overtrue <[email protected]>
|
||
|
|
04b9c8fd36 |
fix(admin): stream madmin ReplicationMRF documents from /v3/replication/mrf (#6126)
* test(admin): pin madmin ReplicationMRF stream contract for /v3/replication/mrf Red-light evidence for backlog#1675 P1-13 (mrf half): madmin's BucketReplicationMRF decodes the response one ReplicationMRF document at a time, so the current aggregate envelope decodes as a single phantom row with an empty object in 'mc replicate backlog'. The new contract tests assert the desired bare-document stream (exact madmin json tags, empty body for an empty backlog) and fail against the current render_mrf_backlog extraction, which preserves the envelope-only behavior: - mrf_stream_renders_bare_madmin_documents: envelope keys leak, no per-entry documents - mrf_stream_renders_empty_body_for_no_entries: empty backlog still renders the envelope (phantom row) - mrf_aggregate_envelope_retains_counters: PerObjectEntriesAvailable never advertises the enumerable stream * fix(admin): stream madmin ReplicationMRF documents from /v3/replication/mrf The mrf endpoint returned a single aggregate envelope, which madmin's json.Decoder loop decoded as one phantom row (empty object) in 'mc replicate backlog' (backlog#1675 P1-13, mrf half; the diff half was fixed in #5799 and this mirrors its pattern). - Default response is now a bare stream of ReplicationMRF documents (exact madmin json tags; Size/TargetARNs as ignored extension keys) built from the durable backlog ledger; an empty backlog renders an empty body, so mc shows zero rows instead of a phantom row. - The aggregate counter envelope moves behind ?aggregate=true (RustFS extension) and now advertises PerObjectEntriesAvailable whenever the durable backlog is readable. - An unreadable backlog is signalled out-of-band via x-rustfs-replication-mrf-backlog-unavailable (mirrors the diff truncation header) plus a warn event, since the bare stream cannot carry source health. - The madmin node parameter is accepted but documented as a no-op: the durable ledger is cluster-shared with no per-node attribution. - Delete-marker purge entries fall back to the marker version id so those rows keep a version identity. * fix(admin): fail the mrf stream request when the durable ledger is unreadable Review: madmin only decodes the body of a 200, so the out-of-band unavailability header was invisible to it and an unreadable ledger read as a clean zero-row backlog. Stream mode now returns 503; aggregate mode keeps the availability fields. * fix(admin): gate, bound, and null-map the mrf stream Second review round: - Authorization: the default stream enumerates object names and version ids, which a metrics-only principal must not see — it now requires admin:ReplicationDiff (MinIO parity, route policy updated); ?aggregate=true carries no object identities and keeps admin:GetReplicationMetrics. - The nil UUID is RustFS's in-memory null-version sentinel and now leaves as the S3 wire token 'null' instead of a zero UUID (a pre-versioning object scanned after versioning + existing-object replication can persist it into the ledger). - The durable ledger is not bounded by the in-memory pending cap and the body is buffered before send; the stream now stops at 10,000 documents and signals truncation via x-rustfs-replication-mrf-truncated (mirroring the diff endpoint) plus a warn event, instead of staging an unbounded body. * fix(admin): reject truncated MRF streams --------- Co-authored-by: Zhengchao An <[email protected]> |
||
|
|
c1f66969d7 |
fix(site-replication): merge incoming ILM expiry documents instead of overwriting (#6130)
* test(site-replication): pin ILM expiry merge contract for incoming lc-config Red-light evidence for backlog#1675 P1-1: the lc-config receiver overwrites the whole local lifecycle config with whatever the peer sends (and deletes it wholesale on peer delete), so an expiry-only document erases the receiver's local tier/transition rules, and peer transition rules get installed across sites. The new tests pin the MinIO mergeWithCurrentLCConfig semantics plus RustFS hardening: - incoming expiry documents merge with (never replace) local rules - local transition sides are authoritative for same-id rules - incoming transition fields are discarded at the trust boundary - dropped expiry rules strip the expiry side but keep transitions; pure-expiry rules are removed - delete merges with the empty set instead of dropping the config - disabled rules survive; abort-mpu-only rules stay site-local - deterministic order (idempotent re-delivery) and expiry_updated_at stamping for the staleness axis All fail against the current overwrite implementation (identity extraction of merge_incoming_lifecycle_config). * fix(site-replication): merge incoming ILM expiry documents instead of overwriting The lc-config receiver replaced the whole local lifecycle config with the peer's document (and deleted it wholesale on peer delete), so an expiry-only update erased the receiver's local tier/transition rules, and a peer's transition rules were installed across sites (backlog#1675 P1-1). Receiver (apply_bucket_meta_item): - lc-config now merges via merge_incoming_lifecycle_config, mirroring MinIO's mergeWithCurrentLCConfig with a trust-boundary hardening: incoming transition fields are discarded outright; the local transition side of a same-id rule is authoritative. A peer delete merges with the empty set — pure-expiry rules go away, transition rules survive with their expiry side cleared, and only an empty result deletes the config file. - Staleness moves to the expiry axis (config.expiry_updated_at): lifecycle_config_updated_at also moves on local transition-only edits, which shadowed newer peer expiry updates. - Receiver-side replicateILMExpiry gate, symmetric with the sender hook (previously any peer could install expiry rules while the option was off). - Rule order is deterministic (local order, incoming-new appended), so re-delivering the same document is byte-stable and does not rewrite bucket metadata per broadcast. Sender: - Both admin choke points — the bucket-meta hook and the SRInfo bucket entry feeding bootstrap/repair and consistency views — now emit only the expiry subset (transition fields stripped, non-expiry rules dropped). MinIO receivers install incoming rules verbatim, so transition rules must never leave the site. An unparseable local config is forwarded unfiltered rather than degraded to a delete. Not covered here (follow-up): a two-site e2e with a real tier backend to exercise transition-rule preservation end to end; receiver-side validate_transition_tier for merged configs. * fix(site-replication): close ILM merge review findings Adversarial review of the lc-config merge surfaced four real defects, all fixed here: - Deletion tombstone regression: with the staleness axis moved to the in-config expiry_updated_at, a deleted lifecycle config fell back to UNIX_EPOCH and any delayed stale broadcast could resurrect deleted expiry rules. The axis now falls back to the whole-config write time (which survives deletion in bucket metadata as the deletion's lower bound), also covering legacy configs that predate the axis field. - MinIO zero-rule documents: MinIO's delete tombstone / transition-only state marshals a lifecycle document with no <Rule>, which the strict s3s deserializer rejects — the receiver now recognizes it as the 'no expiry rules here' statement (delete semantics) instead of erroring on every MinIO heal pass. - Inflated expiry axis at the sender: PutBucketLifecycle stamped expiry_updated_at unconditionally, so a transition-only edit advanced the axis and let this site's stale expiry subset shadow and roll back newer peer expiry edits fleet-wide. The stamp is now conditional (expiry subset present before or after the edit, MinIO parity), the hook item travels with the config's expiry axis (UNIX_EPOCH when the site has none), and the SRInfo bucket entry feeds bootstrap/repair the same axis instead of the whole-config write time. - Del-marker parity: MinIO's CloneNonTransition never emits del-marker or abort-mpu fields, so treating del_marker_expiration as traveling expiry let a MinIO broadcast delete this site's del-marker-only rules. Both fields are now site-local on every edge: stripped from outbound subsets and inbound rules, restored from the local side on same-id merges, and never a deletion criterion. Receiver-side validation of merged configs (object-lock / tier constraints, MinIO runs finalLcCfg.Validate) remains a follow-up. * fix(site-replication): close the second ILM review round - Missed-delete repair: a deleted expiry state now travels through bootstrap/repair as an explicit timestamped lc-config delete item (lifecycle_expiry_statement distinguishes deletion — whole-config write time advanced past the created backfill — from never-configured buckets and from transition-only configs without an expiry axis, which say nothing). A peer that missed the live delete converges on repair; the receiver's staleness guard protects newer peer state. - Strict tombstone recognition: only a well-delimited zero-rule <LifecycleConfiguration> document maps to delete semantics; truncated or foreign payloads that fail the strict deserializer are rejected instead of being treated as a delete that erases local expiry rules. - Staleness fallback axis narrowed: the whole-config write time is used only for deleted or legacy-with-expiry state. A present transition-only config without an expiry axis compares at epoch — its whole-config time moves on transition edits and must not shadow or block independent peer expiry updates and same-timestamp repairs. * fix(site-replication): validate tombstone children structurally Second review round: a well-delimited root could still smuggle malformed content — e.g. <LifecycleConfiguration><ExpiryUpdatedAt> </LifecycleConfiguration> passed the no-<Rule check and was applied as a delete. The tombstone body must now be a sequence of well-formed simple children (matching open/close or self-closing, no nested markup, no stray text, none named Rule); anything else surfaces InvalidRequest. Malformed-child cases pinned in the recognition test. * fix(site-replication): serialize lifecycle merges --------- Co-authored-by: overtrue <[email protected]> |
||
|
|
cfa9276fad |
fix(admin): serialize replication metrics in minio-go wire shapes (#6127)
* test(admin): pin minio-go Metrics/MetricsV2 wire contract for replication metrics Red-light evidence for backlog#1675 P1-11: ?replication-metrics[=2] serializes the internal snake_case BucketStats family straight onto the wire, while minio-go's replication.Metrics/MetricsV2 expect camelCase tags (currStats/queueStats/replicaCount/queued/...). Go's decoder is case-insensitive but does not ignore underscores, so 'mc replicate status' shows all zeros without any error. The rewritten snapshot tests assert the minio-go tags (plus a synthesized queueStats node — the aggregation path leaves queue_stats.nodes empty today) and fail against the current pass-through serialization. * fix(admin): serialize replication metrics in minio-go wire shapes ?replication-metrics[=2] and the admin replicationmetrics endpoint serialized the internal snake_case BucketStats family straight onto the wire, so 'mc replicate status' decoded all zeros without any error (backlog#1675 P1-11). The internal structs cannot be renamed: they are the intra-cluster peer-RPC wire format (rmp_serde to_vec_named in node_service.rs), pinned by a new regression test. - New admin/replication_metrics_wire.rs: Serialize-only projections onto minio-go replication.Metrics (v1 body, currStats) and MetricsV2 (uptime/currStats/queueStats/downtimeInfo) with the exact json tags; per-target failed becomes the TimedErrStats envelope fed from the FailStats rolling window; the queue peak is dual-emitted as max (MinIO server tag) and peak (minio-go tag). - queueStats synthesizes one node from the bucket queue snapshot — the aggregation path leaves queue_stats.nodes empty, and mc treats an empty node list as 'no data' — and carries transfer summaries (Large/Small/Total) derived from the per-target xfer rates. - Both endpoints share the DTOs; source-health extension keys (provider_available/cluster_complete/...) ride along and are ignored by Go decoders. - Widen the ecstore replication_stats_boundary re-exports (BucketReplicationStat/InQueueMetric/XferStats) so the admin facade chain can name the projected types. * fix(replication): carry failure rolling windows through cluster aggregation Review: both metrics endpoints aggregate first, and FailStats::merge dropped the process-local samples (which also never cross the peer-RPC wire — serde-skipped), so lastMinute/lastHour serialized as zero right after a failure while totals was nonzero. - FailStats gains serializable last_minute/last_hour window snapshots (serde default: old nodes read zeros, new fields are ignored by old decoders), recomputed on every add_size and re-stamped at the per-node collection point (get_latest_replication_stats), and summed by merge. - The wire DTO takes the component-wise max of the live samples and the snapshot, so both the single-node and the aggregated path report the window. - Regression test drives a stat through rmp round trip + merge before serialization, as requested. Also restore the #[allow(dead_code)] attribute to route_policy — the new module declaration had been inserted between the attribute and its item, which broke the -D warnings CI lanes. * fix(replication): bin transfer summaries at 128 MiB and keep window refresh off the hot path Second review round: - update_xfer_rate split at 1 MiB while the minio-go transferSummary labels (and RustFS's own worker-pool split) mean >= 128 MiB for Large, so a 2 MiB replication reported under Large with Small stuck at zero. The producer now bins on MIN_LARGE_OBJ_SIZE; a MetricsV2 assertion covers 2 MiB / 127 MiB / exactly 128 MiB. - add_size no longer recomputes the rolling windows: two full one-hour-deque scans per failure under the bucket-stats write lock made failure bursts quadratic (30k events ~2.1s). The windows are stamped only at the collection point (get_latest_replication_stats, which serves both the local leg and the peer RPC); the aggregation regression now drives that path explicitly before the RPC round trip and merge. * fix(replication): average transfer summaries --------- Co-authored-by: overtrue <[email protected]> |
||
|
唐小鸭andGitHub
|
9138c24571 |
fix(site-replication): lift a rejoined site's restarted edit counter over stale marks (#6119)
fix(site-replication): lift a rejoined site's restarted edit counter over stale fence marks A site removed while unreachable (unilateral removal: the receiver never dropped it from its peer map, so parse_site_replication_state's load-time mark pruning never fired) that later rejoins recreates its state object and restarts edit_generation at zero. The receiver's surviving high-water mark then silently fences out every stamped delivery from that origin — peer edits and the add finalize fan-out alike are acked without applying — until the restarted counter catches up. Allocate the generation as a hybrid logical clock instead: max(wall clock in unix nanoseconds, previous + 1), still inside the state transaction under the distributed state-object lock. Every value a lifetime hands out is capped by the wall clock at its own allocation, so a recreated lifetime's first allocation exceeds them all and clears the stale mark, while a pre-removal delivery still in flight stays below the new floor and remains correctly fenced. previous+1 keeps allocations strictly increasing across same-tick allocations and mid-lifetime clock regressions. Nothing changes on the wire or in the persisted schema: editGeneration stays the single fence param and edit_generation the single counter field, so pre-hybrid receivers get the fix as soon as the sender upgrades, old binaries preserve the field across rolling up/downgrades, and marks recorded by plain-counter receivers (small values) are cleared by any wall-clock allocation. A clock that regresses across a delete/recreate degrades to a fence that self-heals once real time passes the previous lifetime's last allocation, and introduces no rollback window beyond what the plain counter already had. An epoch-based design (editEpoch wire param + per-origin epoch marks) was built first and rejected under adversarial review: old binaries rewriting the state object drop the unknown epoch fields, which both disarms the fix mid-rolling-upgrade and — because epoch adoption lowers the generation mark — reopens the pre-restart rollback the fence exists to prevent; a backwards clock also fences an origin permanently instead of self-healing. The hybrid clock has none of these modes. |
||
|
唐小鸭andGitHub
|
e11ce2f132 |
fix(site-replication): route every state RMW through the locked transaction (#6097)
* fix(site-replication): route every state RMW through the locked transaction P1-15 PR2 (rustfs/backlog#1796, batch B2 of rustfs/backlog#1675), the follow-up promised by rustfs/rustfs#5882. PR1 left ~26 read-modify-write call sites on config/site-replication/state.json in the pre-transaction shape: a process-local mutex around load / mutate / save, each IO taking its own object lock. Nothing held a distributed lock across the whole sequence, so two nodes of one site still lost each other's updates, and the transitional mutex kept the old shape available to copy. Every remaining RMW now runs inside update_site_replication_state; read-only sites use load_site_replication_state, whose object read comes with the object-level read lock. SITE_REPLICATION_STATE_LOCK and its owner helper are gone, together with their architecture-guard allowlist entry and inventory row. The multi-stage flows (add / edit / peer join / peer edit / remove / rotate) keep their updated_at and pending-id CAS, but the CAS now runs inside the transaction that writes, against the state that transaction loaded. Peer probes, IAM work and fan-outs run between transactions and hold no lock at all — the add no longer blocks every writer of the site across its peer join round trips, and it re-checks the precondition right after the capability probes so the common race is rejected before any IAM write or remote join. When the add's commit CAS still fails, the error says the peers may already be joined and that re-running the add reconverges. The add adopts only the fields it computed (exhaustive destructure — adding a state field is a compile error until classified); fields owned by writers that do not bump updated_at keep their freshly loaded values. Ordering of peer-edit deliveries now rests on the generation fence landed in PR1 rather than on a guard that could never order two nodes: the add's finalize fan-out carries the generation allocated in its commit. An accepted peer join PRESERVES the applied-generation high-water marks — join fan-outs are routine (adds and rotations both deliver SRPeerJoin to existing peers), so wiping them would let stalled older edits land after any join; the unilateral-removal rejoin misfence that a wipe would have patched is pre-existing since the fence landed and needs an epoch in the fence instead. The rotation handler now takes the lifecycle guard: the background service-account reconciler runs its repair under a lifecycle try-acquire, and its pending-rotation precheck is only sound if a rotation cannot start mid-repair — an exclusion the removed process mutex used to provide as a side effect. update_site_replication_state_when_changed adds persist-or-skip so ack markers and pending-clearing paths stop rewriting the object on a miss — load-bearing, because the shared persist helper clears the whole object for a ≤1-peer pending-free state — and save_site_replication_state is now cfg(test): the pre-P1-15 shape can no longer be written in production code. No on-disk format change. Verification: cargo nextest run -p rustfs -E 'test(/admin::handlers::site_replication::/)' (181 passed); site-replication dual/three-node e2e (13 passed); cargo clippy -p rustfs --all-targets -D warnings; make pre-commit. Mutation checks: dropping the state-object lock from the boundary reds the separate-node concurrency tests; flipping a persist-or-skip miss to a persist reds test_missed_pending_clear_must_not_rewrite_the_state_object. Reviewed by three independent adversarial passes (correctness/concurrency, security/compatibility, simplicity/test-coverage); their confirmed findings are folded in. * fix(site-replication): serialize peer-join admission around its IAM write Review follow-up (overtrue): two joins accepted by the same node could interleave as "A checks a stale snapshot and pauses reading its body, B applies secret B and commits, A resumes, overwrites IAM with secret A, and A's commit is refused as superseded" — the persisted state advertised B's contract while IAM only accepted A's secret, failing every peer control-plane call. The pre-P1-15 process mutex serialized same-node joins end to end; removing it dropped that exclusion. admit_peer_join now runs the staleness check, the IAM upsert and the state commit under the lifecycle guard, with the authoritative pre-check taken against a load under that guard BEFORE IAM changes anything. The closing transaction still re-checks staleness: the guard is process-local (exactly as far as the old mutex reached) and the state-object lock arbitrates joins accepted by different nodes. The body is fully read before the guard so a stalling sender cannot block add/remove/rotate/reconciler. The IAM step is injected, and the gated-body regression test reproduces the review's ordering: join A is held mid-IAM while a newer join B arrives; B must wait at the guard, and both IAM order and the final persisted state end on B. Mutation-verified: removing the lifecycle guard from admit_peer_join turns the test red. Verification: cargo nextest run -p rustfs -E 'test(/admin::handlers::site_replication::/)' (182 passed); site-replication dual/three-node e2e (13 passed); cargo clippy -p rustfs --all-targets -D warnings; make pre-commit. * fix(site-replication): fence peer-join admission across nodes Review follow-up (overtrue, round 2): the lifecycle guard only serializes joins within one process. Node A could pass the staleness check for an older T1, node B write secret B to IAM and commit a newer T2, and node A then overwrite IAM with secret A while its own state commit is refused as superseded — state advertising T2's contract while IAM only accepts A's secret. The admission (staleness check -> IAM upsert -> state commit) now also runs under a distributed join-admission lock, a namespace-lock key with no backing object, following the repair execution lock's pattern — including its nesting of config-object locks (admission -> state), and delegating crash safety to the lock subsystem's lease expiry instead of a hand-rolled TTL. The staleness check runs against a load taken inside the lock, before IAM changes anything, so a superseded join exits without touching IAM. The closing transaction keeps its re-check for defence in depth and for old-version nodes that do not take the admission lock during a rolling upgrade (that mixed-version window keeps today's behavior and closes when the upgrade completes). admit_peer_join_across_nodes is the admission minus the process-local lifecycle guard — exactly what a second node runs — and the new separate-nodes regression test drives it directly with join A gated mid-IAM: join B must wait at the distributed lock, and both the IAM write order and the final persisted state end on B. Mutation-verified: removing the admission lock turns the test red while the same-node test (which drives the full admit_peer_join) stays green. Verification: cargo nextest run -p rustfs -E 'test(/admin::handlers::site_replication::/)' (183 passed); site-replication dual/three-node e2e (13 passed); cargo clippy -p rustfs --all-targets -D warnings; make pre-commit. |
||
|
唐小鸭andGitHub
|
603bdea516 |
fix(site-replication): route state RMW through one locked transaction (#5882)
* test(site-replication): pin retry-event lost-update against locked RMW (red) P1-15 (rustfs/backlog#1675 B2): the site-replication retry-event writers (enqueue/dequeue, which hang off every hook broadcast path) perform a load -> mutate -> persist without taking SITE_REPLICATION_STATE_LOCK, so a single process can lose a concurrent lock-holding writer's update; the service-side reload path is equally unlocked, and no writer holds a distributed lock across the read-modify-write, so multi-node RMW loses updates even where the process lock is held. Red evidence (current main): replaying enqueue's exact three steps around a completed mark_pending_rotation_peer_acked commit wipes the rotation ack — the final state holds the retry event but not the ack. * fix(site-replication): route state RMW through one locked transaction P1-15 PR1 (rustfs/backlog#1675 B2). The site-replication state object (config/site-replication/state.json, which also carries the retry-event queue) was mutated through read-modify-write sequences with inconsistent locking: the retry-event writers on every hook broadcast path and the RPC-driven service reload took no lock at all (single-process lost updates, pinned by the red commit), and no writer held a distributed lock across the whole RMW (cross-node lost updates everywhere). - New admin/site_replication_state module: the state transaction boundary `with_site_replication_state_lock[_on]` — process mutex plus the distributed config-object write lock (the pattern proven by the repair state), with the shared path constant. The process mutex is transitional until PR2 migrates the remaining ~26 call sites. - handlers: typed `update_site_replication_state` (no-lock load / persist-or-clear inside the boundary; normalizes the peer map exactly once, retiring the double-clone/double-normalize persist path, P2-22). Migrated: retry-event enqueue (always-write), dequeue (lock-free probe, transaction on hit), mark_pending_rotation/remove_peer_acked. - service reload: the tolerant byte-level read->normalize->save now runs inside the same boundary via no-lock IO — a cluster-wide reload fan-out can no longer overwrite a concurrent state writer. Normalization semantics untouched (all six service-side tests unchanged and green). - Add/PeerJoin/Edit handlers release the state guard before their peer fan-out: the transport helpers' retry-event bookkeeping now re-enters the state transaction and must not nest inside the guard (the adversarial review caught this as a re-entrancy deadlock; the fix mirrors the Remove/Rotate handlers' existing scope). The Edit non- refresh branch commits before fanning out — the old fanout-first order recorded retry events pointing at a state the local site had not saved. - ecstore: delete_config_no_lock (+ facade/bridge exports) so the clear half of persist-or-clear works under the held object lock. Red -> green: the red commit pinned the deterministic lost-update interleaving (stale retry-event persist wiping a committed rotation ack); the test now drives the real functions concurrently for 8 rounds and asserts every retry event and every ack survives. Full handlers/service site-replication unit suites green (171 + 6); dual-node site-replication e2e (state edit fresh/stale, object replication) green; fmt / clippy / logging guardrails clean. Adversarial review: one blocking finding (the re-entrancy deadlock above) fixed and re-verified by a full second pass over all 30 lock sites and the Add/Join/Edit call graphs. Non-blocking notes recorded for PR2: mark_* now persists on miss (persist-or-clear semantics; a miss-skip return is a cheap follow-up), Add still holds the guard across the peer join probe (pre-existing availability debt), and a timeout-guarded unreachable-peer regression test for the fan-out paths. * fix(site-replication): keep the state mutex behind an owner helper CI's architecture migration guard lists SITE_REPLICATION_STATE_LOCK as an owner-local static, so it may not be `pub(crate)`. Keep it private to the new module and let the not-yet-migrated RMW call sites take it through `site_replication_state_process_guard()` — the sanctioned owner-helper pattern; the helper disappears with the mutex in PR2. * fix(site-replication): keep peer-edit delivery under the state guard Review follow-up (#5882). Releasing the guard before the fan-out (my deadlock fix) traded the ordering the guard used to provide: edit A could commit and stall while edit B committed and reached a peer first, then A arrived last and won. The peer edit handler applies whatever arrives — it has no generation or updated-at fence — and a successful stale delivery is not repaired by the retry queue, so the sites diverge silently. The fan-out is back under the guard. What actually could not run there is the retry-event bookkeeping, which re-enters the state transaction, so the edit branch now delivers with the plain transport and settles the retry queue after the guard is released: successes dequeue, the first failure enqueues and is returned. Ordering and bookkeeping both preserved. The add handler keeps its peer-edit finalize fan-out under the guard for the same reason and releases only before bootstrap/back-fill, which send bucket-ops (not peer edits) through retry-event transports. The concurrency test could not tell the two guards apart — both writers took both locks, so it passed with either removed. Replaced by two tests that isolate one guard each, both verified by mutation: - a process-only legacy writer (the shape the not-yet-migrated call sites still use) racing the transaction: fails when the transaction stops taking the process mutex; - two writers that bypass the process mutex, as separate nodes do, driving the production object-lock path (`with_site_replication_state_object_lock` factored out for exactly this): fails when the distributed lock is removed. Verification: handlers 173 + service 6 unit tests green; site-replication dual-node and three-node edit e2e green; arch/layer/logging guardrails, fmt and clippy clean. * fix(site-replication): fence peer-edit delivery by generation Review follow-up on the two remaining holes in the edit path. Ordering was only process-local. `SITE_REPLICATION_STATE_LOCK` is per node, so holding it across the fan-out orders the edits ONE node accepts and nothing else: two nodes of the same site can both commit and reach a peer in the opposite order, and the peer edit handler applied whatever arrived last. Each edit now takes a generation from `SiteReplicationState::edit_generation`, allocated in the same commit as the edit itself — i.e. under the distributed state-object lock, so two nodes can never share one. The generation rides the peer-edit request as query parameters and the receiver rejects (acks without applying) a delivery at or below the mark it already applied for that origin site, recording the mark in the same commit as the edit it fences. Peers that predate the fence send no parameters and are applied as before. Retry settlement could discard a newer failure. After the guard is released, a success for edit A removed every retry event for (peer, peer-edit): if edit B committed, failed its own delivery and enqueued while A was in flight, A erased it — local state B, peer on A, nothing queued to converge them. Settlement now only removes events whose recorded generation is not newer than the one being settled, and a later failure never lowers the fence. Broadcast paths carry no generation and settle unconditionally as before; their events live under their own paths and cannot collide with a peer-edit delivery. A departed peer's mark is dropped on load: a site that leaves drops below two peers, which clears its state object and restarts its counter at zero, so a leftover mark would reject every edit it sends after it rejoins. Tests: two-node generation uniqueness (drop the object lock and the two nodes collide), the receiver's staleness predicate and its wiring, the settlement interleaving (drop the fence and B's retry is erased), and the rejoin reset. Refs: rustfs/backlog#1675 (P1-15) |
||
|
唐小鸭andGitHub
|
a076ae4045 |
test(replication): pin the scanner existing-object compensation matrix (#5877)
P1-20 (rustfs/backlog#1675 B2, test-only). No prior test wrote objects BEFORE the replication rule arrived, leaving the scanner's existing-object resync pass — the only channel for such objects — without end-to-end coverage, and the enqueue truth table partially unpinned at unit level. e2e (both negative cells are contracts, asserted over multiple fast-scanner cycles next to a replicated control key that proves the scanner and the live path are running): - test_scanner_compensates_existing_objects_across_write_paths: plain PUT, CopyObject and Snowball auto-extract products written pre-rule all converge via scanner compensation; a null-version object (PUT before the bucket became versioned) is pinned as never compensated (the scanner heal gate skips nil-version objects). - test_scanner_never_compensates_when_existing_object_replication_disabled: ExistingObjectReplication=Disabled is a contract, not a delay — existing keys stay absent while post-rule writes replicate normally. Unit truth-table pins (crates/replication): - queue.rs: an empty replicate decision (Disabled existing-object, inbound REPLICA) skips heal queueing for every status; Completed without a resync decision skips. - operation.rs: existing-object resync without a reset replicates exactly the never-replicated (Empty) objects. Helper: put_bucket_replication_with_statuses parameterizes the previously hardcoded ExistingObjectReplication status; the nextest count comments are refreshed to the post-rebase totals. |
||
|
唐小鸭andGitHub
|
2ecf6b4575 |
fix(replication): probe the version-identity contract in replication-check (#5881)
* test(replication): pin the version-fidelity probe contract (red) P1-19 (rustfs/backlog#1675 B2): the supported replication contract is targets that adopt the source version id — a target that mints its own ids silently breaks every version-addressed operation that follows (version deletes, heal re-drives never match), diverging the two sides with no signal. replication-check already captures the probe PUT's response version id but never compares it. Red evidence (current main): against a FakeS3Target with assign_own_version_ids enabled, ?replication-check returns Status "OK" — the drift is invisible. test_replication_check_flags_version_minting_target expects a VersionFidelity phase that fails with the machine-readable code BucketRemoteTargetVersionMismatch, skips the later mutation phases, and still cleans up the probe via the version id the target actually assigned. Test infra: FakeS3Target gains assign_own_version_ids (models a generic S3 service; validated-but-not-mirrored source version headers) and a prefix+max-keys ListObjectVersions implementation (the probe key allocation requires it); stored_versions accessor duplicated from the P1-21 branch (identical code, resolves clean on merge). * fix(replication): probe the version-identity contract in replication-check P1-19 (rustfs/backlog#1675 B2, plan B). Replication only converges on targets that adopt the source version id: version-addressed deletes and heal re-drives address the source id, so a target that mints its own ids silently diverges — nothing surfaced this. replication-check already captured the probe PUT's response version id but never compared it. - The probe PUT now carries the source version as `?versionId=` (the exact shape live replication uses since P0-5, and the only shape MinIO consumes; the internal source-version-id header alone would let the probe pass against targets the real data path drifts on). Reuses ecstore's append_version_id_query through the api facade. - New VersionFidelity phase: the probe PUT's response version id must equal the sent source id. On mismatch the phase fails with the machine-readable extension key `"Code": "BucketRemoteTargetVersionMismatch"` (new optional Code field on phase statuses; Go decoders ignore unknown keys), the overall target fails, the later version-addressed mutation phases are skipped, and cleanup still removes the probe via the id the target actually assigned (with the existing list-based sweep as backstop when the target returns no version id at all). - Runtime half: TargetClient::put_object now returns the assigned version id (mirroring remove_object), and the replication PUT path audits it — every drifting PUT increments rustfs_replication_version_identity_drift_total and the first drift per target ARN logs a structured warning pointing at ?replication-check. The drift judgment is a pure function with an exemption-matrix test (empty / literal "null" / nil-uuid sources carry no contract). - docs/operations/replication-check.md documents the phase and the code. Red -> green: test_replication_check_flags_version_minting_target (fake target with assign_own_version_ids; on main the check reported Status "OK"). The probe's query shape is pinned by a journal assertion (revert of the query hunk alone fails it), probe-level unit tests cover the mismatch/mirror matrix including cleanup addressing the minted id, and the existing success e2e now asserts VersionFidelity OK against a RustFS target. Adversarial review (seven roles): non-blocking; noted follow-ups are the multipart runtime audit (the probe phase already pins the contract) and per-target re-warning after reconfiguration. * fix(e2e): stop the fake target self-deadlocking on version-id minting The assign_own_version_ids flag was read with a fresh `lock(&self.store)` inside two paths that already hold that guard — delete_object's marker-creation branch and create_multipart_upload — and the store mutex is not reentrant, so both hung forever (CI: the fake target's own multipart and delete-marker tests ran >1560s until the job was cancelled). Read the flag from the live guard instead. The replication e2e paths did not catch this: a version-addressed purge DELETE never mints an id, and the probe PUT reads the flag before taking the guard. * chore(test): refresh the nextest replication count invariant The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata (authority: `cargo nextest list`); refresh it to this branch's post-rebase total. |
||
|
唐小鸭andGitHub
|
320b788a50 |
test(admin): relax object-lambda SNI test timeout under full-suite load (#5923)
The SNI preservation test is the only object-lambda test doing a real TLS handshake; the shared helper's 2s whole-request timeout turns concurrent fsync-heavy TestECStoreEnv neighbors into a deterministic TimedOut when the per-build nextest schedule overlaps them. The test verifies SNI, not latency, so widen its budget to a still-bounded 30s. |
||
|
唐小鸭andGitHub
|
3c31eaf06f |
fix(replication): retry, persist and replay failed delete-marker purges (#5864)
* test(replication): pin delayed delete-marker purge failure handling (red) P1-21 (rustfs/backlog#1675 B2): two failing e2e tests that pin the missing failure handling of the delayed delete-marker purge: - test_delayed_delete_marker_purge_retries_after_transient_target_failure: four scripted 503s outlast every existing channel (version-purge replication + its in-process MRF fast retries + the watcher's single attempt = 3 target DELETEs, all faulted in the recorded run); the replicated marker is stranded on the target forever. - test_delayed_delete_marker_purge_exhaustion_persists_to_mrf_and_replays_on_restart: exhausted purge intents never reach the durable MRF journal, so a restart replays nothing (recorded run: 3 faulted attempts, zero post-restart). Red-light evidence (current main): - Test A: FAILED, journal shows 3x DeleteObject fault=Status(503), no clean attempt, target marker still present after 15s. - Test B: FAILED after 468s, same 3 faulted attempts, no purge DELETE after restart, marker still present. Test infra: FakeS3Target::stored_versions() exposes per-key version state so purge tests assert target state instead of inferring it from the journal; nextest count comments 36->38 nightly / 56->58 total. * fix(replication): retry, persist and replay failed delete-marker purges P1-21 (rustfs/backlog#1675 B2). The delayed delete-marker purge was fire-and-forget: the target DELETE discarded its result (`let _ =`), a missing target client was silently skipped, and nothing recorded the intent — one transient target error stranded the replicated marker on the target forever. Separately, `replicate_delete_with_outcome` held its outcome hostage to `!requires_delayed_purge`, pinning every delete-marker MRF entry to Missed so the durable backlog retained them permanently. Changes: - `replicate_delete_marker_purge_to_targets` now reports per-target results (warn + metrics on failure, including `target_client_missing`), supports retrying only the failed targets, and treats a target-side NoSuchKey/NoSuchVersion as purge success (strict-404 targets must not retain the intent forever). - The delayed watcher (`watch_and_purge_source_delete_marker`) retries failed targets across its 5x1s watch window; on exhaustion it persists the purge intent to the durable MRF journal via the new `ReplicationPoolTrait::persist_mrf_entry` (journal-only on purpose: live re-dispatch would loop unboundedly against a down target). Intent entries are shaped as marker-creation deletes so replay funnels into the stale- marker branch. - The stale-marker branch (source marker already gone) now purges the targets instead of silently returning success — closing a latent leak — and reports the purge result as the replay outcome. Heal callers retry for the full window (the startup MRF processor runs before target clients initialize); live callers attempt once and fall back to a fresh durable intent, so a down target cannot pin a replication worker. - The outcome formula (extracted as `replicate_delete_outcome` and pinned by a unit test) no longer includes the delayed purge, so successfully replayed delete-marker entries are acknowledged instead of retained forever. Verification: red -> green e2e pair (transient-failure retry; exhaustion -> durable MRF -> restart replay -> second-restart zero-replay ack) plus unit tests; `make pre-commit`, logging guardrails, clippy (ecstore + e2e_test) all clean; full ecstore lib suite 3729 passed (3 pre-existing local-DNS kubernetes endpoint failures reproduce without this change). Adversarial validation (7 roles): no blocking findings after adding the outcome-formula guard test. Known residuals recorded in the PR: watcher shutdown window (intent not yet persisted), rolling-downgrade replay acks without purging (equals pre-fix behavior), and replay falling back to the source version id on targets that mint their own version ids (P1-19). * chore(test): refresh the nextest replication count invariant The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata (authority: `cargo nextest list`); refresh it to this branch's post-rebase total. * fix(replication): purge the marker version the target actually assigned Review follow-up (#5864), two real defects: - The delayed purge watcher was spawned with the pre-merge `dobj`, so the per-target marker version ids this round recorded were invisible to it. Against a target that mints its own ids the purge fell back to a source-derived id, the target answered the versioned DELETE with an idempotent 204, and that "success" cleared the retry set while the real marker stayed behind. The watcher now receives the merged replication state (`drs`), which folds this round's target-assigned ids in. - A target whose recorded version metadata is inconsistent was skipped without entering `failed_arns`, so an empty result made both the watcher and the MRF replay treat a purge that issued no DELETE as successful and drop the intent. The refusal is now a per-target failure (own metric label): the leak stays visible and the intent is retained instead of being acknowledged. The version decision also moved ahead of the client lookup, so the refusal is decided from metadata alone. Tests: a new e2e drives a fake target with `assign_own_version_ids`, which ignores the forwarded source-version header for both objects and delete markers, and asserts the replicated marker is really gone; a unit test pins the corrupt-metadata refusal as a failed outcome without any target client registered. The detached-watcher shutdown window is documented at the watcher as a known non-durable window with the write-ahead follow-up spelled out. |
||
|
唐小鸭andGitHub
|
88756ea8e1 |
test(ecstore): decouple kubernetes endpoint tests from kernel hostname (#5900)
Three Kubernetes endpoint-identity tests read the real kernel hostname and panicked when it is an IP literal (e.g. macOS without a static HostName, where DHCP/reverse-DNS sets the kernel hostname to an address like 192.168.1.11). Add a cfg(test) override seam (force_kernel_hostname_for_test, mirroring the existing force_local_host_resolution_timeout_for_test pattern) and route the production read through kernel_hostname_for_endpoint_identity() so the tests inject deterministic hostnames instead of depending on the host environment. Production behavior is unchanged. |
||
|
唐小鸭andGitHub
|
6333f21a2e |
feat(replication): SSE-C ciphertext passthrough replication (#5898)
Complete the encrypted-object replication series (backlog#1783, PR-C of 3, after #5872 and #5885): SSE-C objects replicate as ciphertext passthrough — the source holds no customer key, so the stored bytes and their encryption metadata travel verbatim and the replica decrypts only with the original customer key, single-part and multipart. - Sender: SSE-C objects read raw (raw_data_movement_read), transfer at ciphertext size, and range multipart parts over stored part sizes. - Receiver: authorized replication PUTs restore the stored SSE-C keys from the transport headers (exact lowercase forms - the read-path check is case-sensitive), set ObjectOptions.preserve_ciphertext, and skip compression, bucket-default SSE, and sse_encryption behind one restore-derived gate. Multipart uses an internal session marker to store parts verbatim and strips it on complete. - Convergence: the replication HEAD sends x-rustfs-source-replication-check; the target authorizes it as ReplicateObjectAction and skips SSE-C read validation for that request only, so keyless convergence HEADs see etag/size/mtime instead of 400 and SSE-C replicas stop re-driving forever. - e2e: SSE-C contract flips to a key-gated readable replica (no-key and wrong-key GETs fail - the direct silent-plaintext detector); new multipart passthrough contract with ETag/marker/stability assertions. |
||
|
|
73e4ef4dd4 |
feat(replication): replicate managed-SSE objects via target re-encryption (#5885)
Open the managed-SSE replication gate (backlog#1783, PR-B of 3, after #5872): the replication reader already decrypts through the injected object-encryption resolver, so the source sends plaintext plus an encryption intent header (AES256 / aws:kms, never the source key id) and the target re-encrypts on its normal PUT path with its own KMS. No DEK crosses sites. - replication_put_object_options: fail closed only on Unsupported; insert the SSE intent after the strip loop. - TargetClient::create_multipart_upload sends the full opts.header() set, fixing multipart replicas losing content-type/user metadata (plaintext included). - Preserve source ETag and mtime on replicas (authorized replication only): receiver wires x-rustfs-source-etag into preserve_etag for PUT and CompleteMultipartUpload, resolve_complete_etag consumes it, and complete options carry source_etag/source_mtime (absent mtime degrades to epoch, not now_utc). Without this every replication HEAD comparison re-drives re-encrypted objects forever. - e2e: managed SSE contracts flip to success on an independent-KMS dual-process pair (byte-identical plain GET proves target-owned envelopes; ETag/mtime preserved; version stable across scanner cycles; resync converges; multipart keeps structure and metadata); new target-without-KMS fail-closed contract; SSE-C stays FAILED. Co-authored-by: houseme <[email protected]> |
||
|
唐小鸭andGitHub
|
c619d8f2d6 | fix(replication): persist REPLICA status on inbound replication writes (#5878) | ||
|
唐小鸭andGitHub
|
eb377209c1 | docs(ci): make e2e-replication-nightly test-count comments drift-resistant (#5866) | ||
|
唐小鸭andGitHub
|
10c7476883 |
fix(replication): rebuild SSE metadata boundary for encrypted objects (#5872)
Groundwork for encrypted-object replication (backlog#1783, PR-A of 3): - classify_replication_source_encryption: accept the AES256 marker that every stored SSE-C object carries; the SseC arm was unreachable. - Fail closed on sealed material without an SSE marker (MinIO-written objects) instead of replicating ciphertext as plaintext. - Replace the dead VALID_SSE_REPLICATION_HEADERS table with a transport map keyed by the metadata keys the SSE writer actually persists, shared via the new rustfs_utils::http::object_encryption_keys module. - Structurally strip all encryption metadata from outbound replication (x-rustfs-encryption-* envelopes previously passed the filters). - Skip decrypt_checksums for encrypted objects at the boundary so its is_multipart=false (a response-path contract) cannot misroute encrypted multipart objects once managed replication opens. - Redact X-Rustfs-Replication-* SSE transport values in FileInfo Debug. A reconciliation test pins that every key encryption_material_to_metadata produces is either transport-mapped or stripped. All four SSE replication e2e contracts still assert FAILED unchanged. |
||
|
唐小鸭andGitHub
|
6633c80151 |
refactor(kms): close the low-severity follow-ups from the #5668 adversarial re-review (#5817)
* refactor(kms): share the DEK spec mapping and stop re-parsing opened envelopes
- generate_key_material is now the single spec->length mapping for every
backend that mints DEKs itself; the inline copies in the Static and Local
backends are gone, and ChaCha20 (32 bytes, same as AES_256) is accepted
uniformly instead of only by Static.
- The pub(crate) client decrypt of the Local, Vault KV2 and Vault Transit
backends returns (plaintext, master_key_id), so KmsBackend::decrypt no
longer re-parses the envelope it just opened (one JSON parse per SSE GET
instead of two, and unknown-field observability is no longer double-counted).
- Malformed-envelope parse failures now report CryptographicError("parse")
on all backends; Local was the last one mapping them to SerializationError.
- The four KmsBackend::generate_data_key adapters take fields out of
DataKeyInfo instead of cloning, dropping a redundant un-zeroized plaintext
DEK copy and a full ciphertext clone per call; a missing plaintext now
fails closed everywhere instead of returning an empty key on three of four
backends.
* test(kms): pin legacy header fallback, stored-AAD, and decrypt key-id contracts
- a_legacy_aws_kms_object_without_the_cipher_header_still_opens rebuilds the
true pre-internal-header shape (aws:kms mode + S3 key-id header, no
x-rustfs-* headers) and asserts the fallback normalizes the cipher and
re-projects it.
- a_rewritten_sse_c_context_header_fails_authentication is the SSE-C flank of
the stored-AAD tamper check; metadata_without_stored_context_bytes_still_opens
covers the derived-AAD path for both flavours and pins the seal side to the
canonical bytes (mutation-verified).
- data_key_spec_controls_the_length_of_the_generated_key requires every
backend in the matrix to honour all three specs, asserts the envelope
records the requested spec, and round-trips each blob.
- corrupt_ciphertext_fails_cleanly pins unparseable ciphertext to
CryptographicError instead of merely not-InternalError.
- Deleted the never-called assert_validation_error / assert_cryptographic_error
helpers.
|
||
|
唐小鸭andGitHub
|
a0a8eaa0f3 |
fix(storage): reserve internal encryption prefixes in user metadata (#5819)
The write-side filter is_reserved_user_metadata_key only namespaced x-amz-, x-rustfs-internal- and x-minio-internal- keys, while the read-side should_skip_object_metadata_key also strips x-rustfs-encryption-* / x-minio-encryption-* as internal. A client PUT of x-amz-meta-x-rustfs-encryption-algorithm therefore landed on disk as the bare internal key x-rustfs-encryption-algorithm, which the KMS headers_to_metadata path treats as the preferred cipher selector. Not exploitable today (the production decrypt path discards the parsed algorithm and FromStr rejects invalid values), but any future wiring of headers_to_metadata into decryption would hand cipher choice to the client. Reserve both encryption prefixes on the write side so client-supplied keys are namespaced under x-amz-meta- like other reserved keys, hoist the prefix constants to module scope shared with the read-side skip logic, and pin the attack form (header injection and CopyObject REPLACE metadata), the bare-header form, and the legitimate server-written SSE metadata flow with regression tests. |
||
|
唐小鸭andGitHub
|
3792fed827 |
fix(replication): madmin reset/diff wire compat and config validation (#5799)
* fix(admin): align replication-reset responses with madmin ResyncTargetsInfo shape
The replication-reset and replication-reset-status responses serialized
their shell as "Targets" and per-target fields in PascalCase, while
madmin-go ResyncTargetsInfo/ResyncTarget expect the "target" shell key
and lowercase field tags (arn/resetid/resyncStatus/replicationCount/
completedReplicationSize/failedReplicationCount/failedReplicationSize).
Go json decoding is case-insensitive per field, but Targets vs target,
Status vs resyncStatus and the size/count key names cannot match, so
mc replicate resync decoded empty results.
Rename the serde tags to the exact madmin wire shape, keep the
ResetBeforeDate/Error RustFS extension keys (unknown keys are ignored
by Go decoders), pin the shape with a snapshot unit test, and update
the e2e client DTO to decode the madmin shape.
* fix(admin): stream bare madmin DiffInfo documents from replication diff
POST /v3/replication/diff returned a single enveloped object
({Entries, IsTruncated, ScannedVersions}) while madmin-go
BucketReplicationDiff decodes the body with a json.Decoder loop over
bare DiffInfo documents. The envelope decoded as exactly one DiffInfo
with an empty object, so mc replicate diff printed a phantom empty row
instead of the real backlog.
Emit one DiffInfo JSON document per line by default, using the exact
madmin json tags (object/versionId/rStatus/deletemarker/lastModified;
Size stays as a RustFS extension key that Go decoders ignore). The
enveloped shape moves to the opt-in ?aggregate=true RustFS extension,
which remains the only carrier of scan-coverage metadata; a truncated
default-mode scan is surfaced via a warn tracing event instead of
in-stream. Pin both shapes with unit tests and tighten the e2e helper
to reject any envelope in the stream.
* feat(replication): validate replication config structure before persisting
PutBucketReplication accepted structurally invalid configurations that
MinIO's replication.Config.Validate rejects: empty or oversized rule
lists, duplicate or negative rule priorities, over-long rule IDs,
filters carrying more than one of Prefix/Tag/And, and delete marker
replication enabled on tag-filtered rules. Such configs persisted
silently and later produced undefined routing (e.g. ambiguous priority
ties) instead of failing the PUT.
Add validate_replication_config_structure as a pure function in
rustfs-replication (limits documented as constants), surface it through
the ecstore api facade, and run it first in the PUT capability gate so
defects are named before any metadata write. Missing Priority counts as
zero for the uniqueness check, matching Go's zero-value semantics. The
self-target rejection deliberately stays at set-remote-target, where the
endpoint is known; a config can never reference a self-pointing ARN.
Document the rule-level Destination.StorageClass contract (use the
remote target's storage_class instead) and renumber the acceptance
matrix e2e to unique priorities, which MinIO would also require.
* test(replication): pin duplicated wire types with boundary reconciliation tests
rustfs-filemeta (xl.meta disk format) and rustfs-replication (MRF/resync
persistence format) deliberately each own ReplicationStatusType,
VersionPurgeStatusType and ReplicationState; the boundary converts
between them via as_str(), whose From<&str> impls fall back to Empty on
unknown tokens — a variant added on one side silently degrades to Empty
on the other.
Add reconciliation tests in replication_filemeta_boundary: exhaustive
matches with no wildcard arm on both sides of both enums (a new variant
fails compilation until the mapping is reconsidered), string-token
round-trip asserts (a token the other side does not recognize fails
instead of quietly becoming Empty), and a full-field ReplicationState
round-trip. Cross-reference the tests from both type definitions.
Struct drift was already compile-guarded by the exhaustive struct
literals in the conversion functions.
* docs(replication): define split completion criteria and milestone sequence
The ecstore replication split plan had no completion measure — the
boundary scaffolding risked ossifying because nothing said when the
migration counts as done. Record the criteria in the module inventory:
done means the Required Contracts table's 'Current dependency to
remove' column is empty; the end state moves pool/resyncer/state into
crates/replication, with the boundary micro-files dissolving as code
crosses the crate line (batch-merging them beforehand is explicitly
rejected — the guard scripts anchor on their file names, so merging is
churn with zero functional gain; only datatypes.rs can retire early).
Sequence the remaining work as M2 (resyncer pure decision logic, after
the oversized function splits) → M3 (worker runtime, highest risk,
last) → M4 (retire boundaries and guard entries). Refresh the stale
first-step text — the event sink / runtime contracts already landed —
and update the split-plan status table accordingly.
* fix(replication): align structural validator with MinIO semantics after adversarial review
Three interop corrections found by adversarial review of the new
structural validator, plus review fallout fixes:
- Delete-marker replication is now rejected only for a direct Filter.Tag,
not for tags inside Filter.And — MinIO's validator only inspects the
direct tag, and mc replicate add --tags "k1=v1&k2=v2" (delete-marker
replication on by default) puts multiple tags into And.Tags, so the
stricter check rejected mc-generated configs MinIO accepts.
- Rule ID length is measured in bytes (Go len semantics), not chars —
a 255-char multibyte ID must not round-trip into a config MinIO
rejects.
- An empty <Tag/> element (no key) counts as absent, matching MinIO's
Tag.IsEmpty(); console form serializers emit empty tags, which would
otherwise trip the exactly-one-of and delete-marker checks.
Also: repair the store-uninitialized PUT test whose empty-rules fixture
now (correctly) fails structural validation before reaching the store
lookup; pin the previously untested startTime madmin key in the
reset-status shape test; and signal a truncated default-mode diff scan
via the x-rustfs-replication-diff-truncated response header — the bare
madmin stream has no envelope, so a truncated scan was otherwise
indistinguishable from a complete healthy one (madmin/mc ignore unknown
headers).
* test(e2e): activate SSE-S3 replication contract and pin resync fail-closed path
The SSE-S3 replication contract e2e was ignored under backlog#1291
(silent plaintext replication); the fail-closed gate in
replication_target_boundary.rs closed that hole, so the ignore reason
expired. Un-ignore the test — it now pins the current fail-closed
contract (FAILED status, failure event, readable encrypted source,
stable absence of all target versions), verified green.
Add test_bucket_replication_sse_s3_resync_stays_fail_closed: drives the
existing-object resync path (PUT ?replication-reset) over a FAILED
SSE-S3 object and asserts the resync generation reaches a terminal
state without ever materializing a target version, with the
stays-absent window also spanning fast-scanner heal cycles. The new
start_bucket_replication_reset helper doubles as the madmin
ResyncTargetsInfo shape assertion (target[0].arn/resetid) for the
reset-start response.
Refresh the stale nextest count commentary (the module is at 20 fast +
36 nightly = 56 tests by cargo nextest list; the SSE-S3-ignored note no
longer holds).
|
||
|
唐小鸭andGitHub
|
8d582a096c |
fix(replication): tolerate Go zero-value expiration and ignore latency in remote target requests (#5789)
* test(replication): accept real madmin marshal payload with zero-value expiration * fix(replication): tolerate Go zero-value expiration and ignore latency in remote target requests |
||
|
唐小鸭andGitHub
|
10abef4791 |
fix(ecstore): parse ARN region and id in display order (#5790)
* test(ecstore): pin ARN display/parse round-trip field order * fix(ecstore): parse ARN region and id in display order |
||
|
唐小鸭andGitHub
|
434663f2aa |
fix(replication): report remote target latency as Go duration nanoseconds (#5771)
madmin-go decodes LatencyStat.curr/avg/max as Go time.Duration (nanosecond integers), but the list-remote-targets admin response serialized them via the persisted milliseconds encoding, so mc showed latency values shrunk by 10^6 (e.g. 50ms rendered as 50ns). Extend remote_target_admin_json — the same response-only re-encode path already used for healthCheckDuration/totalDowntime — to emit the latency stats as nanoseconds, leaving the persisted bucket-targets wire format (milliseconds) untouched. list_targets overwrites latency from live health stats before serialization, so the response path is the single conversion point. Found by the MinIO compatibility review (P2). |
||
|
唐小鸭andGitHub
|
6303aa9a42 |
fix(site-replication): translate policy mapping userType at MinIO wire boundary (#5751)
* test(site-replication): pin MinIO IAMUserType wire semantics for policy mappings Red tests for P0-4: MinIO peers send SRPolicyMapping.UserType using the madmin IAMUserType table (unknown=-1, regUser=0, stsUser=1, svcUser=2), while RustFS deserializes the field as u64 and decodes it with the internal RPC table (None=0, Svc=1, Sts=2, Reg=3). - userType -1 (MinIO group mappings) fails to deserialize, rejecting the whole IAM item: group mappings never sync from MinIO. - stsUser=1 decodes as Svc, landing federated STS mappings under the wrong prefix and silently dropping their effect. * fix(site-replication): translate policy mapping userType at MinIO wire boundary SRPolicyMapping.userType travels on the wire using MinIO's IAMUserType table (unknown=-1, regUser=0, stsUser=1, svcUser=2), but RustFS stored the field as u64 and reused the internal RPC encoding UserType::to_u64/from_u64 (None=0, Svc=1, Sts=2, Reg=3) at the site replication boundary. Consequences: MinIO group mappings (userType -1) failed to deserialize and the whole IAM item was rejected, and MinIO STS mappings (1) were stored as service-account mappings, silently dropping federated users' policies. - Widen SRPolicyMapping.user_type and SRCredInfo.iam_user_type to i64 so MinIO's -1 deserializes. - Add sr_wire_user_type / user_type_from_sr_wire in rustfs-iam as the dedicated SR wire codec: MinIO table on both directions, groups always encoded as 0, and wire value 3 kept forever as an alias for Reg so mappings from pre-fix RustFS peers still decode; unknown values fail closed. - Route the SR inbound (apply_iam_item) and outbound (mapped_policy_to_sr_mapping, policy-mapping change hooks) paths through the codec. The internal UserType::to_u64/from_u64 encoding is untouched: it is the intra-cluster node RPC contract and changing it would break rolling restarts. Outbound compatibility with old RustFS peers is preserved because UserType::None and Reg share the users prefix in get_mapped_policy_path, so wire 0 lands in the same location Reg=3 did. |
||
|
唐小鸭andGitHub
|
dbf51117a1 |
fix(replication): schedule replication for CopyObject and snowball extracted objects (#5753)
* test(replication): expect CopyObject and snowball extract to schedule replication
Red-phase TDD tests for P0-6: CopyObject never consults the bucket
replication config (no pending stamp, no schedule, and the destination
inherits the source's stale replication status metadata wholesale), and
snowball auto-extract members are never scheduled either.
- usecase white-box: observe MUST_REPLICATE_OBJECT_CALLS for
execute_copy_object (currently 0, must be 1) and
execute_put_object_extract (currently 0, must be 2 for a two-member
archive), plus stale replication-status metadata cleanup assertions
(MinIO filterReplicationStatusMetadata parity).
- e2e: CopyObject destination and snowball-extracted members must appear
on the remote replication target and reach COMPLETED on the source.
Red evidence (before fix):
copy_object_computes_replication_decision_and_strips_stale_status
assertion failed: left: 0, right: 1
put_object_extract_computes_replication_decision_per_entry
assertion failed: left: 0, right: 2
* fix(replication): schedule replication for CopyObject and snowball extracted objects
CopyObject and snowball auto-extract never consulted the bucket
replication config: no PENDING stamp, no post-commit schedule, and no
scanner-heal backstop (heal only re-drives Pending/Failed objects, and
these objects carried no status at all). Worse, the copy path cloned the
source metadata wholesale, so a destination object inherited the
source's replication bookkeeping and could present a fake
COMPLETED/REPLICA state.
Mirroring the PUT path (single immutable decision drives both the
pending metadata and the post-commit schedule, rustfs/backlog#1320):
- execute_copy_object: strip the source's replication status metadata
(internal replication/replica status + timestamps under both
compatibility prefixes, plus x-amz-replication-status) for
non-inbound requests — MinIO filterReplicationStatusMetadata parity;
the cleanup runs before the decision so an inherited REPLICA status
cannot suppress it. Then compute must_replicate_object once, stamp
PENDING when it replicates, and schedule after the copy commits and
the self-copy lock guard is released. Inbound replica writes keep
their authorized metadata and are declined inside
must_replicate_object, so replicas are never re-scheduled outbound.
- execute_put_object_extract: same stamp + schedule per extracted
member object (MinIO PutObjectExtract parity).
- execute_put_object dispatch: an authorized inbound replication PUT is
stored verbatim instead of being re-dispatched into the extract path.
Extracted members keep x-amz-meta-snowball-auto-extract in their user
metadata and the replication client replays stored metadata as
headers, so the target used to try to untar each member's own bytes,
permanently failing replication for non-archive members (surfaced by
the new snowball e2e test).
Green evidence:
- copy_object_computes_replication_decision_and_strips_stale_status,
put_object_extract_computes_replication_decision_per_entry (red: 0
decisions; green: 1 and 2), plus the existing PUT/object-lock
decision-count tests stay green.
- e2e test_copy_object_replicates_to_target and
test_snowball_extract_replicates_members_to_target pass against two
live instances.
|
||
|
唐小鸭andGitHub
|
923e35efa0 |
fix(site-replication): use MinIO-compatible sts-account IAM item type (#5750)
* test(site-replication): expect MinIO sts-account IAM item type
MinIO madmin-go replicates STS credentials with SRIAMItem type
"sts-account" (SRIAMItemSTSAcc), but RustFS emits and accepts only
"sts-credential", so cross-implementation STS replication fails in
both directions (MinIO returns errSRInvalidRequest, RustFS returns
NotImplemented).
Red-light tests:
- pin the outbound AssumeRole replication item type to "sts-account"
(construction extracted into assume_role_site_replication_item so it
is testable, behavior unchanged in this commit)
- update the federated identity replication item snapshot to
"sts-account"
- inbound apply_iam_item must dispatch both "sts-account" and the
legacy "sts-credential" alias to the STS arm instead of the
unknown-type NotImplemented fallback
* fix(site-replication): use MinIO-compatible sts-account IAM item type
MinIO madmin-go replicates STS credentials with SRIAMItem type
"sts-account" (SRIAMItemSTSAcc). RustFS emitted "sts-credential" and
accepted only that value inbound, so STS credential replication with
MinIO peers failed in both directions: MinIO rejected RustFS items as
errSRInvalidRequest and RustFS answered MinIO items with
NotImplemented.
- define SR_IAM_ITEM_STS_ACC ("sts-account") and
SR_IAM_ITEM_STS_ACC_LEGACY ("sts-credential") in rustfs-madmin
- emit "sts-account" from both outbound sites (AssumeRole hook and
federated identity OIDC hook)
- accept both types inbound; the legacy alias remains permanently for
mixed-version RustFS rolling upgrades
Token verification and the retry/event mechanism are unchanged.
|
||
|
唐小鸭andGitHub
|
733c7b0f67 |
fix(replication): accept remote target healthCheckDuration nanoseconds (#5754)
* test(replication): accept madmin nanosecond healthCheckDuration payloads Red-phase TDD tests for P0-7: mc 'replicate add' sends the madmin default healthCheckDuration=60s as a Go time.Duration nanosecond integer (60000000000), which RustFS currently rejects as an unsupported field and would misread as seconds. Also pins the defensive seconds-or-nanos read for persisted bucket-targets metadata and the capability contract listing healthCheckDuration as writable. Currently failing (red): - remote_target_request_accepts_go_duration_wire_values - remote_target_request_accepts_legacy_seconds_health_check - remote_target_health_check_duration_is_declared_writable - bucket_target_reads_go_nanosecond_durations_defensively - runtime_capabilities_response_reports_missing_topology_before_storage_init * fix(replication): accept remote target healthCheckDuration nanoseconds mc 'replicate add' always sends the madmin default healthcheck-seconds=60 serialized as a Go time.Duration nanosecond integer (60000000000), so the default mc link-creation path (and 'mc replicate update') failed with InvalidRequest. Move healthCheckDuration from the unsupported to the writable remote-target field list; the capability contract in the runtime capabilities response follows the constants automatically. Fix the unit mismatch in both directions: - Request parsing and persisted bucket-targets reads decode the value defensively: below 10^7 it is legacy RustFS seconds, otherwise Go time.Duration nanoseconds (also covers MinIO-written metadata). totalDowntime shares the same wire shape and gets the same handling. - The list-remote-targets admin response re-encodes only these two fields as nanoseconds via a dedicated serialization path, leaving the persisted seconds-based wire format untouched for existing readers. The per-target health-check interval is accepted for mc compatibility but not yet applied; the heartbeat keeps its global env-configured interval, and the explicit 'healthcheck' update op stays rejected. disableProxy, edge, and edgeSyncBeforeExpiry remain explicitly rejected. |
||
|
唐小鸭andGitHub
|
ead419451a |
fix(replication): send source versionId as query param to remote targets (#5752)
* test(replication): assert remote PUT and multipart initiate carry versionId query * fix(replication): send source versionId as query param to remote targets |
||
|
唐小鸭andGitHub
|
04722caa04 |
fix(madmin): accept MinIO PascalCase SRInfo fields and nil-map nulls (#5749)
* test(madmin): add MinIO PascalCase SRInfo fixture coverage * fix(madmin): accept MinIO PascalCase SRInfo fields via serde alias * test(madmin): cover MinIO nil-map SRInfo JSON output * fix(madmin): tolerate Go nil-map null in SRInfo deserialization |
||
|
唐小鸭andGitHub
|
066e952df1 |
fix(site-replication): send peer join to MinIO peer/join route with encrypted payload (#5748)
* test(site-replication): pin peer join wire path to MinIO peer/join route MinIO only ever registered PUT /minio/admin/v3/site-replication/peer/join; the /site-replication/join path never existed upstream. Flip the wire-path and payload-encryption expectations to the real MinIO route. These tests fail until the outbound rewrite is fixed. * fix(site-replication): send peer join to MinIO peer/join route with encrypted payload MinIO only registers PUT /minio/admin/v3/site-replication/peer/join; the /site-replication/join path never existed upstream, so the outbound join special-case rewrote requests to a 404 route. Drop the special case so peer join falls into the generic /rustfs -> /minio prefix rewrite, and move the payload-encryption predicate to the peer/join route (MinIO's SRPeerJoin force-decrypts the request body). MinIO also replies with an empty body on a successful join, which the previous strict JSON parse rejected. Tolerate an empty/whitespace body by synthesizing the peer identity from the add preflight metainfo (deployment id) already fetched for the site. Inbound dual-path registration (join and peer/join under both admin prefixes) is intentionally unchanged for rolling upgrades from older RustFS peers that still send the legacy outbound path. |
||
|
唐小鸭andGitHub
|
ea8dbf49a2 |
fix(auth): route ListBuckets denial through ApiError to hold s3s ratchet (#5755)
fix(auth): route ListBuckets auth denial through ApiError to keep s3s ratchet at baseline PR #5726 added one s3_error! call in authorize_request while PR #5739 froze the s3_error! line baseline at 1686 counted before that merge, so a clean main-derived branch fails the s3s footprint ratchet with +1. Replace the new macro call with ApiError::access_denied().into(), a small constructor on the gateway-side error abstraction (rustfs/backlog#1677 F1, rustfs/backlog#1733) instead of raising the baseline. The converted S3Error carries the identical AccessDenied code and "Access Denied" message, and the filtered ListBuckets fallback matches on the code only. |
||
|
|
15b9c1f4e3 |
fix(replication): make bucket replication rules editable from clients (#5715)
* fix(replication): accept explicit STANDARD destination storage class The replication engine never reads Rule.Destination.StorageClass (replica placement comes from the bucket-target config or the source object), yet the validator rejected any config carrying the field. The console's add-rule form always sends StorageClass=STANDARD, so every rule created through it failed with InvalidRequest. Tolerate exactly STANDARD as a no-op — semantically identical to omitting the field — and keep rejecting every other value, which would be silently ignored rather than honored. Document the deliberate omission from the replication capability contract. * feat(admin): support MinIO-style partial updates for set-remote-target set-remote-target?update=true previously replaced every stored field and required complete credentials in the body, so flipping a target's sync mode from the console forced operators to re-enter the secret key, and real mc replicate update bodies (madmin Clone() strips the secret) failed to deserialize at all. Adopt MinIO's TargetUpdateType contract: query params creds/sync/bandwidth/ path name the field groups to overlay onto the stored target, everything else keeps its persisted value, and unsupported groups (proxy, healthcheck, edge, edgeSyncBeforeExpiry) fail loudly. Credentials updates are skipped for site-replication peer targets — probed by both scheme derivations of the stored endpoint and the stored deployment id — because an operator never knows the site replicator's credentials, and a body-supplied deployment id is ignored on update since it anchors peer identity. madmin JSON aliases (bandwidthlimit, storageclass, resetID, deploymentID, sessionToken) let mc bodies parse under deny_unknown_fields. e2e: cover a credential-free sync-only update preserving the stored connection and the zero-ops no-op contract; align the missing-arn assertion with the earlier validation error. * chore(scripts): add two-site replication lab manager site_replication_smoke.py spawns and manages two local rustfs processes, pairs them via the site-replication admin API (idempotent), and verifies bidirectional object replication. Subcommands: up/down/restart/status/logs/ smoke/info/remove/clean. Stdlib-only; requests are SigV4-signed the same way as crates/e2e_test. * chore(scripts): rename direction-suffixed payload variables for typos check The typos linter reads the _ba suffix in payload_ba as a misspelling of "by"; use payload_a_to_b / payload_b_to_a instead. --------- Co-authored-by: overtrue <[email protected]> |
||
|
|
62cc19e937 |
fix(kms): repair unopenable ciphertext and cover the Vault backends (#5668)
* Add black-box behavior tests for KMS resilience and serialization * fix(kms): repair unopenable ciphertext across backends Black-box testing of the KMS crate surfaced several defects that make encrypted data permanently unreadable. Symmetric envelopes. The Local and Vault Transit backends returned raw cipher output from `encrypt` while `decrypt` parsed a JSON envelope, so anything sealed through the master-key path could never be opened again. Local also discarded the AES-GCM nonce. Both now emit the same envelope `decrypt` consumes, matching the Static backend. Deterministic AAD. The object layer derived AEAD additional data by serializing a `HashMap` directly. Iteration order differs per instance, so a context rebuilt from storage produced different AAD bytes than the one used to seal and the object stopped opening. Ordering by key removes that dependency, matching the Static backend's existing `context_aad`. Objects written with the default single-key context are unaffected, since a one-entry map has only one serialization. Cipher in the header projection. `metadata_to_headers` recorded the SSE mode (`AES256` / `aws:kms`), which cannot represent ChaCha20-Poly1305, so a ChaCha-sealed object came back claiming `aws:kms` and was opened with the wrong cipher. The cipher now travels in `x-rustfs-encryption-algorithm` — the header the storage layer already reads but nothing ever wrote. Objects without it fall back as before. Also: the Static backend ignored `key_spec` and always issued 256-bit data keys; Local `list_keys` hardcoded `truncated: false`, ignored `marker`, and paginated over unordered `read_dir`, so a paginating client silently saw a partial key list; and Local and Vault KV2 reported `key_id: "unknown"` from `decrypt` despite the envelope naming the master key. Co-Authored-By: Claude Opus 5 <[email protected]> * test(kms): cover both Vault backends and key rotation The behavior suite ran only against Local and Static, and its own harness documented the gap: the Vault backends had no business-capability coverage at all. Setting `RUSTFS_KMS_VAULT_TOKEN` now adds Vault KV2 and Vault Transit to every `for_each_backend` spec against a live server. That lane is what surfaced the Transit envelope defect fixed in the previous commit. `rotate` and `versioning` are advertised only by the Vault backends, so until now every capability-gated branch for them took the `UnsupportedCapability` side and the working half was never asserted — a rotation that dropped prior key versions would have gone green. The new `behavior_rotation.rs` pins that half: material sealed before a rotation still opens after it, repeated rotations accumulate versions rather than overwriting a single spare, and the history survives a restart. Two test defects fixed. `objects_round_trip_across_sizes_and_algorithms` asserted a 1-byte object differs from its own ciphertext, which collides once every 256 runs; the assertion now applies only where a collision is not realistic, and small objects stay covered by the tag check and the decrypt round-trip. `test_from_env_selects_token_file` depended on `RUSTFS_KMS_VAULT_TOKEN` being absent from the caller's environment and now clears it explicitly. The snapshots directory was also removed from `.gitignore`: insta snapshots are the assertions themselves, so leaving them untracked gives CI nothing to compare against. Only `.snap.new` scratch files are ignored now. Co-Authored-By: Claude Opus 5 <[email protected]> * test(kms): adapt behavior suite to current key APIs Rebasing onto main brought four API changes the suite predates. `DeleteKeyRequest` gained `confirm_key_id`, and immediate deletion is now gated on the server's `allow_immediate_deletion`. Scheduled deletions pass `None`; the four specs that destroy a key outright echo the key id back and opt the harness config in, which is what the gate asks of a real caller. `LocalBackupExportRequest` gained `sanitized_config`. These specs cover the key-material path, so they seal no configuration and pass `None`. `KmsCacheStats` became a named struct with real hit, miss, and eviction counters. `cache_stats_returns_an_entry_count_and_no_hit_or_miss_data` existed to pin the old placeholder behavior — that the second tuple element was always zero — which main has since fixed, so it is now `cache_stats_reports_hits_and_misses_separately` and asserts the counters actually move. Starting the service provisions the reserved probe key, so it shows up in listings and backup bundles. Exact-set assertions filter it through a new `without_probe_key` helper rather than naming it, keeping those specs about the keys they seeded. Co-Authored-By: Claude Opus 5 <[email protected]> * fix(kms): bind the AAD to the stored context bytes Review caught that canonicalizing the AAD on decrypt breaks objects sealed before canonicalization existed, and it was right. The AAD is the *serialization* of the encryption context, and `x-rustfs-encryption-context` stores that exact byte sequence: `encrypt_object` fed one `HashMap` to the AEAD and then moved the same map into the metadata the header is written from, so the stored string is byte-identical to the AAD the object was sealed under. Those objects are therefore recoverable — but only while nothing round-trips the value through a `HashMap` and re-serializes it. Recomputing sorted AAD on decrypt would have turned a readable object into a permanently unreadable one. The previous behavior was worse than the first analysis credited: it did not merely fail intermittently, it made the failure deterministic. `EncryptionMetadata` now carries `context_aad`, the bytes the object was actually sealed with. Encryption records what it fed the AEAD, the header projection stores those bytes verbatim (and preserves a legacy ordering across a re-projection rather than rewriting it into sorted form), and `headers_to_metadata` carries the stored string through untouched. Both decrypt paths, SSE-KMS and SSE-C, prefer it and fall back to canonical serialization only when no stored serialization exists. Canonicalization still applies to everything newly sealed, so the original ordering bug cannot recur. Two tests pin this: a legacy record whose sealed bytes are non-canonical must survive a full header round trip unchanged, and a context header rewritten to an equivalent-but-reordered serialization must fail authentication rather than silently re-deriving a working AAD. Both were mutation-checked against the reinstated bug on each side. Also from review: the lifecycle churn test asserted only that every request was accounted for, which holds whether the state gate exists or not, so both branches are now pinned deterministically after the churn (asserting `refused > 0` on the concurrent phase would only trade the hole for a scheduling flake). And the Local and Vault KV2 envelopes compare `encryption_context` without authenticating it — `DekCrypto` seals only the plaintext — which is now documented at both sites; closing it needs a versioned envelope, since existing ciphertext was sealed without AAD. Co-Authored-By: Claude Opus 5 <[email protected]> --------- Co-authored-by: Claude Opus 5 <[email protected]> |
||
|
唐小鸭andGitHub
|
3f716746cf | fix(replication): honor target TLS in health checks (#5613) | ||
|
|
c8016cbcdb |
fix(replication): enforce bucket replication switches (#5449)
* fix(replication): enforce bucket replication switches * fix(replication): satisfy delete admission clippy lint * fix(replication): restore MinIO tag filter behavior --------- Co-authored-by: Zhengchao An <[email protected]> Co-authored-by: cxymds <[email protected]> |
||
|
|
ad7663afd1 |
refactor(sse): decouple ecstore and harden KMS lifecycle (#5435)
* refactor(sse): decouple encryption from ecstore * feat(kms): enhance KMS service manager with runtime state and persistence support * feat(kms): add local key export functionality for SSE-S3 migration tests * fix(kms): keep local key export narrowly scoped * fix(sse): validate copy source customer algorithm --------- Co-authored-by: Zhengchao An <[email protected]> |
||
|
|
f329d330df |
feat(kms): support safe local KMS evaluation workflows (#5418)
* feat(kms): enable safe local KMS evaluation workflow * test(kms): align SSE reconfigure coverage --------- Co-authored-by: cxymds <[email protected]> |
||
|
唐小鸭andGitHub
|
2216f00cfd |
fix(kms): unify persisted SSE data key envelopes (#5343)
* feat(kms): implement secure handling of static KMS secret keys and enhance encryption context validation * feat: enhance local SSE DEK handling with JSON envelope format and versioning |
||
|
唐小鸭andGitHub
|
99e1f5fbd2 | feat(kms): add static single-key backend (#5222) | ||
|
唐小鸭andGitHub
|
233865d172 | feat(kms): introduce KMS unavailability error and enhance data key handling (#5184) | ||
|
唐小鸭andGitHub
|
d13345dc65 |
fix(site-replication): preserve HTTPS peer join endpoints (#5045)
fix(site-replication): enhance TLS handling for peer joins and add tests |
||
|
唐小鸭andGitHub
|
7ab1abee80 |
fix(admin): allow site replication peer edits (#4623)
* fix(site-replication): align IAM and bucket metadata replication * fix(admin): allow site replication peer edits |
||
|
唐小鸭andGitHub
|
849ea9a122 | fix(site-replication): align IAM and bucket metadata replication (#4318) | ||
|
唐小鸭andGitHub
|
188ab2131d | fix(kms): persist Vault Transit key metadata (#4262) | ||
|
唐小鸭andGitHub
|
4efefd67b5 |
fix(site-replication): site replication flag issue and resolved the replication storm (#4120)
* fix(site-replication): Add Docker Compose setup for site replication testing - Fixed the site replication flag issue and resolved the replication storm. - Introduced a new directory for site replication tests with Docker Compose. - Created `docker-compose.yml` to define three RustFS sites and a setup container. - Added `README.md` to document the purpose, usage, and test flow of the replication setup. - Implemented `run-object-flow-check.sh` script to verify object replication across sites. - Configured health checks and volume permissions for the RustFS containers. - Enabled customization of access keys, bucket names, and other parameters via environment variables. * fix * fix |
||
|
|
eff656e086 |
fix(storage): restore legacy SSE-S3 read compatibility (#3584)
* Update .gitignore * Fix. fixed SSE-S3 compatibility issues in large-scale testing * fix * fix(ecstore): reject whitespace bucket names * Update replication_extension_test.rs * style(ecstore): format bucket whitespace test --------- Co-authored-by: houseme <[email protected]> Co-authored-by: cxymds <[email protected]> |
||
|
|
8afb963d21 |
feat(ecstore): optimize the triggering conditions of the compression module (#3387)
feat. Optimize the triggering conditions of the compression module Co-authored-by: houseme <[email protected]> |
||
|
|
f7724d223b |
feat(rio): rio_v2 is compatible with minio for storing data. (#3115)
* Set up a compatibility layer for replacing old Rio components with new ones. * fix(rio). compress range * feat(rio). Add the experimental feature rio_v2 to support minio data at the binary level. * feat(rio_v2): add sse-c test * test compression component * simple fix * fix minlz encode * fix metadata * fix kms key cache error * Update launch.json * ci: set nix crate download user agent * fix: gate obs pyroscope backend * ignore minio test * fix encrypt check * fix * fix * fix * Update object_usecase.rs * Update ci.yml * fix * ci add rio-v2 test * fix * ci fix * fix * Reconstructed into a more reasonable compatibility mode * fix * fix --------- Signed-off-by: houseme <[email protected]> Signed-off-by: 唐小鸭 <[email protected]> Co-authored-by: houseme <[email protected]> Co-authored-by: cxymds <[email protected]> Co-authored-by: 安正超 <[email protected]> |
||
|
|
74296761fa |
fix(replication): repair site replication setup (#3252)
* fix: repair site replication setup * Optimized site status collection --------- Co-authored-by: cxymds <[email protected]> |
||
|
唐小鸭andGitHub
|
480babc0af |
fix: window error (#3167)
fix window error |
||
|
|
09c2d15057 |
fix(sse): Temporarily refactored the SSE design for ECStore (#2813)
Co-authored-by: houseme <[email protected]> Co-authored-by: cxymds <[email protected]> |
||
|
|
e16f1ae639 |
fix(window): Compatible with Windows Path (#2691)
Co-authored-by: houseme <[email protected]> |
||
|
|
d949d4e794 |
fix: avoid sending HEAD bodies over TLS HTTP/2 (#2648)
Signed-off-by: 唐小鸭 <[email protected]> Signed-off-by: houseme <[email protected]> Co-authored-by: Copilot Autofix powered by AI <[email protected]> Co-authored-by: houseme <[email protected]> Co-authored-by: loverustfs <[email protected]> |
||
|
|
4aafb07173 |
refactor: update binary field types and conversions in RPC and protofiles (#2619)
Signed-off-by: 唐小鸭 <[email protected]> Co-authored-by: Copilot Autofix powered by AI <[email protected]> Co-authored-by: houseme <[email protected]> Co-authored-by: loverustfs <[email protected]> |
||
|
|
fb0d096d5d |
fix(sse). Resolving Nonce Overwriting Issues in Multi-Package Scenarios (#2582)
Signed-off-by: 唐小鸭 <[email protected]> Co-authored-by: houseme <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: copilot-swe-agent[bot] <[email protected]> |
||
|
|
72928a43fa |
Update rustfs/src/admin/handlers/kms_dynamic.rs
Co-authored-by: Copilot <[email protected]> Signed-off-by: 唐小鸭 <[email protected]> |
||
|
|
55f175b671 |
Update rustfs/src/admin/handlers/kms_dynamic.rs
Co-authored-by: Copilot <[email protected]> Signed-off-by: 唐小鸭 <[email protected]> |
||
|
|
3a16c25cd9 |
Update crates/kms/src/backends/vault_transit.rs
Co-authored-by: Copilot <[email protected]> Signed-off-by: 唐小鸭 <[email protected]> |
||
|
reatang
|
30a757a9da | feat(kms): add vault transit engine | ||
|
|
f89cdfe5b3 |
update s3s 0.14.0-dev (#2070)
Co-authored-by: houseme <[email protected]> |
||
|
唐小鸭andGitHub
|
fff96a0921 | fix sse-options (#2056) | ||
|
|
f17725a2ea |
fix(sse): allow PUT/GET without KMS when no SSE or bucket default (#2054)
Co-authored-by: houseme <[email protected]> |
||
|
|
568c07ced9 |
fix: implement handling for "aws-chunked" Content-Encoding (#2009)
Co-authored-by: 安正超 <[email protected]> |
||
|
唐小鸭andGitHub
|
4413878739 | fix(compress): downgrade non-compressible log level and expand excluded file types (#1780) | ||
|
|
7a42af922d |
Refactor: refactor SSE layer and KMS subsystem (#1703)
Co-authored-by: houseme <[email protected]> |
||
|
|
add6453aea |
feat: add seek support for small objects in rustfs (#1231)
Co-authored-by: loverustfs <[email protected]> |
||
|
|
1057953052 |
fix: Remove the compression check that has already been handled by tower-http::CompressionLayer. (#1190)
Co-authored-by: houseme <[email protected]> Co-authored-by: loverustfs <[email protected]> |