* fix: preserve S3 listing pagination parity
S3 listing responses need stable wire semantics across encoded listings, version markers, and quorum-sensitive metadata reads. This tightens marker handling, response encoding, and version-marker pagination while keeping the changes scoped to listing paths and regression tests.
Constraint: Branch, code, commit, and PR text must avoid restricted upstream project naming.
Constraint: Verification required Rust 1.95 toolchain path because Homebrew cargo-clippy resolved to 1.94.
Rejected: Treat null version-id-marker as no marker | repeats or skips the marker boundary for null-version listings.
Rejected: Compare encoded next-marker values directly | encoded output can diverge from raw listing order.
Confidence: high
Scope-risk: moderate
Directive: Do not change listing marker semantics without covering V1, V2, versions, encoded responses, and null-version markers together.
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit with LC_ALL=en_US.UTF-8 and Rust 1.95 PATH
Tested: git diff --check
Not-tested: live distributed object-store compatibility test against a remote cluster
* fix: keep listing echo fields raw
S3 compatibility tests expect list response echo fields such as Prefix, Delimiter, StartAfter, and Marker to preserve the request value even when URL encoding is requested. Keep URL encoding scoped to object keys and common prefixes while preserving the next-marker raw comparison fix.
Constraint: PR branch was updated with latest main before this fix.
Constraint: Do not use restricted upstream project naming in commit text.
Rejected: Encode all response string fields | breaks compatibility tests for unreadable prefix values.
Confidence: high
Scope-risk: narrow
Tested: cargo test -p rustfs list_objects --lib
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit
Tested: git diff --check
Not-tested: live CI s3 compatibility rerun before push
* fix: accept empty listing continuation token
S3 compatibility tests treat an empty ListObjectsV2 continuation token as an explicit empty echo value, not as an invalid base64 token. Preserve the request echo while skipping decoded-token pagination for the empty string.
Constraint: Keep invalid non-empty continuation tokens rejected before store lookup.
Rejected: Drop empty continuation tokens from the response | compatibility tests assert the empty echo field is present.
Confidence: high
Scope-risk: narrow
Tested: cargo test -p rustfs list_objects --lib
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit
Tested: git diff --check
Not-tested: live CI s3 compatibility rerun after this push
* fix: delete explicit null object versions
S3 compatibility cleanup lists unversioned objects with VersionId=null and then sends that value back through DeleteObjects. The API layer keeps null as an internal sentinel, but the storage delete path must map it back to the stored null version instead of treating it as a real UUID.
Constraint: Keep the wire response able to echo null version IDs.
Rejected: Treat VersionId=null the same as an absent version id everywhere | versioned and suspended buckets need explicit null version semantics.
Confidence: high
Scope-risk: moderate
Tested: cargo test -p rustfs-ecstore delete_file_info_version_id_maps_explicit_null_version_to_stored_null
Tested: cargo test -p rustfs normalize_delete_objects_version_id_preserves_explicit_null_marker --lib
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit
Tested: git diff --check
Not-tested: live CI s3 compatibility rerun after this push
* fix: keep version marker scoped to marker key
Version listing cleanup can delete the key returned as the previous page marker before requesting the next page. When the marker key is no longer present, the version marker must not be applied to the first later key, otherwise each cleanup page can skip one null-version object and leave the bucket non-empty.
Constraint: Preserve version-marker pagination for marker keys that still exist.
Rejected: Drop version markers whenever the marker key was supplied | multi-version marker keys still need intra-key pagination.
Confidence: high
Scope-risk: narrow
Tested: cargo test -p rustfs-ecstore version_marker_is_applied_only_when_key_marker_entry_is_present
Tested: cargo test -p rustfs-ecstore delete_file_info_version_id_maps_explicit_null_version_to_stored_null
Tested: cargo test -p rustfs normalize_delete_objects_version_id_preserves_explicit_null_marker --lib
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit
Tested: git diff --check
Not-tested: live CI s3 compatibility rerun after this push
---------
Co-authored-by: houseme <[email protected]>
Co-authored-by: loverustfs <[email protected]>
Object repair rebuilt missing data shards but left missing parity shards unrecreated. A parity-only repair could therefore complete without restoring the shard contents.
Add a repair-specific reconstruction path that regenerates parity after data shards are available, while keeping normal read decoding data-only. Also shut down repair writers after all blocks are written.
Constraint: Preserve read-path decode behavior and limit parity regeneration to repair.
Confidence: high
Scope-risk: moderate
Directive: Do not route read decoding through parity regeneration without measuring the cost.
Tested: cargo test -p rustfs-ecstore decode_data_and_parity -- --nocapture
Tested: cargo test -p rustfs-ecstore heal_reconstructs_missing_parity_shard -- --nocapture
Tested: cargo test -p rustfs-ecstore erasure_coding -- --nocapture
Tested: cargo test -p rustfs heal_object_marks_missing_shard_disk_dirty_for_capacity_manager -- --nocapture
Tested: cargo test -p rustfs-heal -- --nocapture
Tested: cargo clippy -p rustfs-ecstore --all-targets -- -D warnings
Tested: cargo fmt --all --check
Tested: make pre-commit
Not-tested: Live multi-node disk replacement outside local test harness
* Preserve multipart replication recovery state
Multipart uploads previously only scheduled replication after completion, leaving no persisted pending state for scanner recovery if the initial async work was lost. Persist the same pending replication metadata during multipart initialization and let completion evaluate the object metadata that was actually stored.
The scanner heal path also treated ordinary pending objects as delete-replication candidates. Restrict that path to delete markers and version purge state so pending objects remain eligible for object replication heal.
Constraint: Bucket replication recovery depends on persisted object metadata after the async queue is unavailable.
Rejected: Rely only on immediate completion-time scheduling | it cannot recover after process restart or worker loss.
Confidence: high
Scope-risk: moderate
Directive: Keep multipart upload initialization aligned with single PUT replication metadata semantics.
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 make pre-commit
Tested: Runtime replication outage check confirmed multipart xl.meta stores PENDING status and timestamp.
* fix(replication): preserve version purge scanner state
Role-derived replication configs need target-scoped status strings before scanner heal can build per-target purge status. The duplicated replication-status assignment left version purge status unset, so scanner recovery could lose the target-level purge state.
Constraint: Scanner heal derives per-target purge decisions from version_purge_status_internal.
Rejected: Leave the duplicate as a harmless cleanup | it changes recovery behavior for role-only configs with version purge state.
Confidence: high
Scope-risk: narrow
Directive: Keep role-derived replication and version purge internal status mapping symmetric.
Tested: cargo fmt --all --check
Tested: cargo test -p rustfs-ecstore heal -- --nocapture
---------
Co-authored-by: wly <[email protected]>
Co-authored-by: houseme <[email protected]>
* fix: keep scanner walk timeouts from offlining drives
Scanner walk operations can time out on large or slow directory listings without proving the backing drive is faulty. Keep the timeout error local to the scan while preserving failure marking for ordinary disk operations.
Constraint: Scanner walk_dir can include listing work that exceeds the drive timeout under slow storage.
Rejected: Disable timeout failure marking globally | real stuck disk operations must still affect drive health
Confidence: high
Scope-risk: narrow
Directive: Do not route scanner/listing timeout back into drive offline state without reproducing issue #2651
Tested: cargo test -p rustfs-ecstore timeout
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit with en_US.UTF-8 locale
Related: https://github.com/rustfs/rustfs/issues/2651
* fix: keep remote scanner walks from offlining drives
Remote walk_dir uses a streaming request that can hit total or stall timeouts during large scans without proving the remote drive is faulty. Route that scanner path through an explicit ignore action while preserving failure marking for ordinary remote operations.
Also treat Duration::ZERO as no operation timeout for remote health tracking, matching the existing local disk wrapper contract while still marking network-like errors on default paths.
Constraint: Scanner walk_dir streams can be slow because of directory size or consumer backpressure.
Rejected: Disable remote timeout failure marking globally | normal remote disk operations still need to evict failed connections and update drive health
Confidence: high
Scope-risk: narrow
Directive: Do not reintroduce remote scanner timeout failure marking without reproducing issue #2651 against remote disks.
Tested: cargo test -p rustfs-ecstore execute_with_timeout
Tested: cargo test -p rustfs-ecstore timeout
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit with en_US.UTF-8 locale
Related: https://github.com/rustfs/rustfs/issues/2651
* test: prove scanner walk backpressure keeps drives online
Add a local scanner walk regression test where the output writer never makes progress. The test confirms the walk timeout returns without marking the drive faulty, covering a stall source that is not caused by object count or network transfer speed.
Constraint: Scanner walk can block on downstream writer backpressure as well as disk or network IO.\nConfidence: high\nScope-risk: narrow\nTested: cargo test -p rustfs-ecstore walk_dir_writer_backpressure_timeout_does_not_mark_drive_failure\nTested: cargo test -p rustfs-ecstore timeout\nTested: cargo fmt --all --check\nTested: cargo clippy --workspace --all-features --all-targets -- -D warnings\nTested: make pre-commit
---------
Co-authored-by: houseme <[email protected]>
* fix: unblock empty chunked admin rebalance start
Some admin clients send empty POST requests with chunked framing and no explicit content length. The admin rebalance route carries no request body, so normalize that known route before downstream validation sees a missing length.
Constraint: Keep normalization scoped to known empty-body admin routes
Rejected: Relax transfer-encoding handling for S3 routes | broader protocol risk
Confidence: high
Scope-risk: narrow
Tested: cargo test -p rustfs --lib server::layer::tests::
Tested: cargo fmt --all --check
Tested: git diff --check
* fix: keep rebalance listing from stalling
Large object migration was awaited inside the listing callback, so metacache readers could stop being drained while drive walk operations were still writing listing data. Move entry migration into bounded background tasks and wait for them after each set listing completes.
Constraint: Preserve existing rebalance cancellation and first-error propagation
Rejected: Only increase drive walk timeouts | masks callback backpressure and still fails for slower migrations
Confidence: medium
Scope-risk: moderate
Tested: cargo test -p rustfs-ecstore rebalance_unit_tests
Tested: cargo test -p rustfs-ecstore cache_value::metacache_set::tests
Tested: cargo test -p rustfs --lib server::layer::tests::
Tested: cargo clippy -p rustfs-ecstore --lib --all-features -- -D warnings
Tested: cargo clippy -p rustfs-ecstore --tests --all-features -- -D warnings
Tested: cargo fmt --all --check
Tested: git diff --check
---------
Co-authored-by: loverustfs <[email protected]>
Co-authored-by: houseme <[email protected]>
Co-authored-by: cxymds <[email protected]>
