Compare commits

..
Author SHA1 Message Date
housemeandheihutu a814bf3ae3 merge: resolve conflicts with main branch
- Merge origin/main into perf/fileinfo-optimization
- Resolve conflicts in crates/ecstore/src/set_disk/ops/object.rs
- Keep AHashMap import and main branch's detailed imports

Co-Authored-By: heihutu <[email protected]>
2026-08-27 16:03:35 +08:00
housemeandheihutu d628a2f48b feat(filemeta): fix tests for AHashMap adaptation
- Import AHashMap in metacache.rs
- Fix metacache_entry_with_mod_time to use AHashMap
- Fix metacache_entry_with_erasure_versions to use AHashMap
- Fix metacache_entry_single_version to use AHashMap
- Fix make_file_info_with_metadata to convert HashMap to AHashMap
- Fix object_part_info_strategy to use AHashMap for checksums
- Fix file_info_strategy to use AHashMap for metadata
- Fix legacy_version_body_round_trips_through_encode to use AHashMap

All 260 tests pass. AHashMap adaptation complete.

Refs: https://github.com/rustfs/backlog/issues/2005

Co-Authored-By: heihutu <[email protected]>
2026-08-26 23:34:33 +08:00
housemeandheihutu 2bd1d70729 feat(ecstore): complete AHashMap adaptation
- Add ahash dependency to ecstore crate
- Import AHashMap in object.rs and bucket_lifecycle_ops.rs
- Update StaleMultipartUploadCandidate.metadata to use AHashMap
- Update stale_upload_lifecycle_due to use generics
- Update stale_upload_current_size_with_opts to use generics
- Fix all .into() calls to use iter().collect() for AHashMap conversion
- Fix user_defined assignments to use iter().collect()
- Fix replacement_metadata to use AHashMap

All compilation errors resolved. rustfs-ecstore now compiles successfully.

Refs: https://github.com/rustfs/backlog/issues/2005

Co-Authored-By: heihutu <[email protected]>
2026-08-26 22:18:20 +08:00
housemeandheihutu 0b96a992d6 feat(ecstore): continue AHashMap adaptation (partial)
- Update merge_replication_metadata_lww to use generics
- Update restore_metadata_update_preserves_protected_metadata to use generics
- Update has_encrypted_part_layout_marker to use generics
- Update clean_metadata, clean_metadata_keys, remove_standard_storage_class to use generics
- Update update_hash_quorum_metadata_map, update_hash_target_delete_marker_versions to use generics
- Fix fi.metadata assignment to use .into()
- Fix lookup call to use get method
- Fix replacement_metadata to use AHashMap

Note: There are still 14 compilation errors remaining in ecstore.

Refs: https://github.com/rustfs/backlog/issues/2005

Co-Authored-By: heihutu <[email protected]>
2026-08-26 21:56:51 +08:00
housemeandheihutu bbc96c43a2 feat(ecstore): update functions to support AHashMap (partial)
- Update restore_operation_id_from_metadata to use generics
- Update require_restore_operation_id to use generics
- Update restore_commit_operation_id_from_metadata to use generics
- Update should_persist_encryption_original_size to use generics
- Update strip_internal_multipart_metadata to use generics
- Update multipart_bucket_incarnation_id to use generics
- Update multipart_bucket_incarnation_matches to use generics
- Update validate_multipart_bucket_incarnation to use generics
- Update tier_destination_id_from_metadata to use generics
- Update get_raw_etag to use generics

Note: This is a partial implementation. There are still compilation
errors in ecstore that need to be fixed.

Refs: https://github.com/rustfs/backlog/issues/2005

Co-Authored-By: heihutu <[email protected]>
2026-08-26 21:45:04 +08:00
housemeandheihutu ec9aabcf00 feat(fileinfo): use AHashMap for metadata fields
- Add ahash dependency to workspace, filemeta, and utils crates
- Change FileInfo.metadata to AHashMap<String, String>
- Change ObjectPartInfo.checksums to Option<AHashMap<String, String>>
- Change MetaObjectV1.meta to AHashMap<String, String>
- Change MetaObjectV1Part.checksums to Option<AHashMap<String, String>>
- Change UniquePartChecksums to use AHashMap
- Make metadata_compat functions generic over BuildHasher
- Make get_internal_replication_state generic over BuildHasher

This optimization replaces the standard library's SipHash with ahash,
which provides 2-3x faster hashing for typical key types.

Refs: https://github.com/rustfs/backlog/issues/2005

Co-Authored-By: heihutu <[email protected]>
2026-08-26 21:36:43 +08:00
59 changed files with 343 additions and 3507 deletions
@@ -6,7 +6,7 @@ description: "Run the end-to-end RustFS console gate, version bump, preview vali
This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. That Release is temporary: `build.yml` deletes it automatically once the final tag's Release is published, so the Releases page ends up carrying deliverables only while the `-preview.N` tags stay behind as the traceability record. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
Pipeline shape:
@@ -19,7 +19,6 @@ check console main against its latest Release
-> validate with latest rc client
-> report preview acceptance results -> STOP for explicit human confirmation
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
-> CI deletes the <target>-preview.N Releases (tags kept)
```
On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
@@ -52,16 +51,14 @@ Rules:
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
## Hard rules
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
- Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
- Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
- Completing preview acceptance does not authorize the final tag. After Phases 35 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
@@ -233,7 +230,6 @@ git push origin "<target>"
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
## Output contract
@@ -243,5 +239,5 @@ Always report:
- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
- Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
- Any deviation from this pipeline and why the user approved it.
-1
View File
@@ -5,4 +5,3 @@ self-hosted-runner:
- sm-standard-2
- sm-standard-4
- dind-sm-standard-2
- smoke-testing
-49
View File
@@ -1033,55 +1033,6 @@ jobs:
echo "🎉 Released $TAG successfully!"
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
# Remove the internal preview releases once the deliverable release is live.
# Only the Releases are deleted; the -preview.N tags stay so the validated
# commit remains traceable.
cleanup-preview-releases:
name: Cleanup Preview Releases
needs: [ build-check, publish-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: write
steps:
- name: Delete preview releases for this target
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
set -euo pipefail
TAG="${{ needs.build-check.outputs.version }}"
RELEASES_JSON="${RUNNER_TEMP}/releases.json"
# Fetch before filtering: a failed listing must abort here instead of
# looking like "nothing to clean up".
gh api --paginate "repos/${GITHUB_REPOSITORY}/releases?per_page=100" > "$RELEASES_JSON"
# Match only <target>-preview.<digits>. String operations, not a
# regex over the tag, so dots in the version cannot widen the match.
DELETED=0
while IFS= read -r preview_tag; do
[[ -n "$preview_tag" ]] || continue
echo "🧹 Deleting preview release $preview_tag (tag kept)"
gh release delete "$preview_tag" --yes
DELETED=$((DELETED + 1))
done < <(
jq -r --arg tag "$TAG" '
.[]
| select(.tag_name | startswith($tag + "-preview."))
| select(.tag_name | ltrimstr($tag + "-preview.") | test("^[0-9]+$"))
| .tag_name
' "$RELEASES_JSON"
)
if [[ "$DELETED" -eq 0 ]]; then
echo "️ No preview releases to clean up for $TAG"
else
echo "✅ Removed $DELETED preview release(s) for $TAG"
fi
alert-on-failure:
name: Alert on scheduled failure
needs: [build-check, prepare-platform-matrix, build-rustfs, build-summary]
-126
View File
@@ -1,126 +0,0 @@
name: RustFS Heal Test
on:
workflow_dispatch:
inputs:
package_url:
description: 'Direct .deb URL (nightly/R2). Defaults to the latest nightly deb.'
required: false
type: string
stop_node_gb:
description: 'Stop the outage node when surviving nodes reach N GiB'
required: false
default: '15'
warp_stop_gb:
description: 'Stop warp when surviving nodes reach N GiB'
required: false
default: '40'
heal_target_gb:
description: 'Outage node must reach N GiB after heal to pass'
required: false
default: '40'
cleanup_before:
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
type: boolean
default: true
cleanup_after:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
permissions:
contents: read
# Only one test at a time: both this and the pool-expansion workflow mutate
# the same test environment, so they share one concurrency group.
concurrency:
group: rustfs-pool-expansion-test
cancel-in-progress: false
defaults:
run:
shell: bash
env:
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
RUSTFS_API_ENDPOINT: ${{ secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }}
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
jobs:
heal-test:
runs-on: smoke-testing
timeout-minutes: 480
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Show environment
run: |
uname -a
jq --version
openssl version
warp --version || true
df -h /data | tail -1
- name: Reset test environment (before)
if: ${{ inputs.cleanup_before != 'false' }}
run: |
chmod +x scripts/test/rustfs_heal_test.sh
./scripts/test/rustfs_heal_test.sh --reset -y
- name: Install RustFS package & start cluster
run: |
ARGS=(--steps "1,2" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./scripts/test/rustfs_heal_test.sh "${ARGS[@]}"
- name: Preflight checks
run: |
ARGS=(--preflight --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./scripts/test/rustfs_heal_test.sh "${ARGS[@]}"
- name: Run heal test (write -> outage -> heal -> verify)
run: |
./scripts/test/rustfs_heal_test.sh \
--steps "3,4,5,6,7" -y \
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
--stop-node-gb "${{ inputs.stop_node_gb }}" \
--warp-stop-gb "${{ inputs.warp_stop_gb }}" \
--heal-target-gb "${{ inputs.heal_target_gb }}" \
--log-file /tmp/rustfs-heal-test.log
- name: Upload test logs
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: rustfs-heal-test-${{ github.run_id }}
path: |
/tmp/rustfs-heal-test*.log
/tmp/rustfs-warp.*.log
if-no-files-found: warn
- name: Reset test environment (after)
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
./scripts/test/rustfs_heal_test.sh --reset -y
- name: Notify on failure
if: failure()
run: |
echo "RustFS heal test failed"
echo "Package source: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
echo "See the uploaded log artifact for details."
+8 -97
View File
@@ -30,18 +30,6 @@ on:
description: 'Run the pool decommission step (3-pool topology only)'
type: boolean
default: true
stop_node_gb:
description: 'Heal: stop the outage node when surviving nodes reach N GiB'
required: false
default: '15'
warp_stop_gb:
description: 'Heal: stop warp when surviving nodes reach N GiB'
required: false
default: '40'
heal_target_gb:
description: 'Heal: outage node must reach N GiB after heal'
required: false
default: '40'
cleanup_before:
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
type: boolean
@@ -50,10 +38,9 @@ on:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
workflow_run:
# Run after the nightly build completes: pool expansion first, then heal.
workflows: ["Nightly GNU Build"]
types: [completed]
schedule:
# Nightly regression run; remove if you do not want a schedule.
- cron: '0 21 * * *'
permissions:
contents: read
@@ -74,23 +61,19 @@ env:
RUSTFS_API_ENDPOINT: ${{ secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }}
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
# Package used by the nightly run (workflow_dispatch inputs are empty for
# workflow_run events), i.e. the latest nightly deb published by nightly-gnu.yml.
# Package used by the scheduled run (workflow_dispatch inputs are empty for
# schedule events), i.e. the latest nightly deb published by nightly-gnu.yml.
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
jobs:
pool-expansion-test:
runs-on: smoke-testing
timeout-minutes: 360
# Run on manual dispatch, or when the nightly build completed successfully
# (its deb is what the tests install). Skipped when nightly failed.
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
ref: ${{ github.event.workflow_run.head_sha || github.ref }}
- name: Show environment
run: |
@@ -108,7 +91,7 @@ jobs:
- name: Install RustFS package & start first pool
run: |
ARGS=(--steps "1,2,3" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
ARGS=(--steps 1,2,3 -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
elif [ -n "${{ inputs.rustfs_version }}" ]; then
@@ -154,8 +137,8 @@ jobs:
with:
name: rustfs-pool-test-${{ github.run_id }}
path: |
/tmp/rustfs-pool-test*.log
/tmp/rustfs-warp.*.log
/tmp/rustfs-pool-test.log
/tmp/rustfs-warp.log
if-no-files-found: warn
- name: Reset test environment (after)
@@ -169,75 +152,3 @@ jobs:
echo "RustFS pool expansion test failed"
echo "Package source: ${{ inputs.package_url || inputs.rustfs_version || 'nightly (R2 latest)' }}"
echo "See the uploaded log artifact for details."
# Heal regression runs after the pool test regardless of its outcome: a pool
# failure must be reported (it makes the run red) but must not block heal.
heal-test:
name: Heal test (after pool test)
runs-on: smoke-testing
timeout-minutes: 480
needs: pool-expansion-test
if: ${{ always() && (github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success') }}
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
ref: ${{ github.event.workflow_run.head_sha || github.ref }}
- name: Reset test environment (before)
if: ${{ inputs.cleanup_before != 'false' }}
run: |
chmod +x scripts/test/rustfs_heal_test.sh
./scripts/test/rustfs_heal_test.sh --reset -y
- name: Install RustFS package & start cluster
run: |
ARGS=(--steps "1,2" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./scripts/test/rustfs_heal_test.sh "${ARGS[@]}"
- name: Preflight checks
run: |
ARGS=(--preflight --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
else
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
fi
./scripts/test/rustfs_heal_test.sh "${ARGS[@]}"
- name: Run heal test (write -> outage -> heal -> verify)
run: |
./scripts/test/rustfs_heal_test.sh \
--steps 3,4,5,6,7 -y \
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
--stop-node-gb "${{ inputs.stop_node_gb || '15' }}" \
--warp-stop-gb "${{ inputs.warp_stop_gb || '40' }}" \
--heal-target-gb "${{ inputs.heal_target_gb || '40' }}" \
--log-file /tmp/rustfs-heal-test.log
- name: Upload test logs
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: rustfs-heal-test-${{ github.run_id }}
path: |
/tmp/rustfs-heal-test.log
/tmp/rustfs-warp.*.log
if-no-files-found: warn
- name: Reset test environment (after)
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
./scripts/test/rustfs_heal_test.sh --reset -y
- name: Notify on failure
if: failure()
run: |
echo "RustFS heal test failed"
echo "See the uploaded log artifact for details."
Generated
+4
View File
@@ -91,6 +91,7 @@ dependencies = [
"const-random",
"getrandom 0.3.4",
"once_cell",
"serde",
"version_check",
"zerocopy",
]
@@ -9626,6 +9627,7 @@ dependencies = [
name = "rustfs-ecstore"
version = "1.0.0-rc.4"
dependencies = [
"ahash",
"arc-swap",
"async-channel",
"async-recursion",
@@ -9771,6 +9773,7 @@ dependencies = [
name = "rustfs-filemeta"
version = "1.0.0-rc.4"
dependencies = [
"ahash",
"arc-swap",
"byteorder",
"bytes",
@@ -10811,6 +10814,7 @@ dependencies = [
name = "rustfs-utils"
version = "1.0.0-rc.4"
dependencies = [
"ahash",
"base64-simd",
"blake2",
"brotli",
+3
View File
@@ -368,6 +368,9 @@ hotpath = { version = "0.24.0", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.48" }
# High-performance hashing
ahash = { version = "0.8", default-features = false, features = ["std", "runtime-rng", "serde"] }
[workspace.metadata.cargo-shear]
ignored = ["hotpath", "rustfs"]
-8
View File
@@ -40,14 +40,6 @@ pub const HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS: u64 = 5_000;
pub const ENV_HEALTH_CLUSTER_TIMEOUT_MS: &str = "RUSTFS_HEALTH_CLUSTER_TIMEOUT_MS";
pub const DEFAULT_HEALTH_CLUSTER_TIMEOUT_MS: u64 = 2000;
/// Timeout for one remote lock-client online check used by readiness (milliseconds).
///
/// This is intentionally shorter than the generic lock RPC timeout so
/// `/health/ready` can report degradation instead of riding a dead peer's
/// connect or HTTP/2 keepalive budget.
pub const ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS: &str = "RUSTFS_HEALTH_LOCK_ONLINE_TIMEOUT_MS";
pub const DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS: u64 = 1000;
/// Maximum time to wait for local node runtime readiness (storage / IAM / lock
/// quorum) during startup before failing fast (seconds).
///
-33
View File
@@ -288,39 +288,6 @@ pub const DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 0;
const _: () = assert!(!DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE);
/// Enable automatic foreground admission for large or unknown-size PutObject requests.
///
/// Unlike the strict experimental gate above, this default-on path only applies
/// to requests that are large enough to create sustained erasure/RPC pressure.
/// Small PUTs continue on the legacy path unless the strict gate is explicitly
/// enabled.
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE";
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: bool = true;
/// Maximum large foreground PutObject requests admitted concurrently per process.
///
/// `0` derives a conservative default from the local disk-read scheduler cap,
/// currently clamped to protect the commit path without making ordinary high
/// throughput uploads single-file.
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT";
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: usize = 0;
/// Minimum object size that enters automatic large PutObject admission.
///
/// Requests with an unknown size are treated as large because the write pressure
/// cannot be bounded from headers.
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES";
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 32 * 1024 * 1024;
/// Time in milliseconds a large foreground PutObject waits for a permit.
///
/// A short wait smooths transient bursts while still returning S3
/// `SlowDown`/503 before body ingest when the node is already saturated.
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 250;
const _: () = assert!(DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE);
/// Environment variable for minimum GetObject timeout in seconds.
///
/// When dynamic timeout calculation is enabled, this is the minimum timeout
@@ -1,220 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Anonymous access to SSE-KMS objects under per-key authorization.
//!
//! Locks both halves of the anonymous contract decided in backlog#2028 (D4):
//!
//! - **Enforcement on**: anonymous requests hold no `kms` grants, so a public
//! bucket policy does not let them read SSE-KMS objects or write through an
//! SSE-KMS default-encryption rule. Both fail with `AccessDenied`.
//! - **Enforcement off** (the default): bucket policy alone governs anonymous
//! access, matching the pre-enforcement behavior — public SSE-KMS objects are
//! decrypted and served, and anonymous writes are encrypted under the default
//! key.
//!
//! The denial today is emergent — an empty-account principal falling through to
//! the IAM default deny — so without this file a refactor of principal
//! construction or policy evaluation could silently flip it. Each test carries a
//! plaintext-object positive control: a denial proves nothing while the bucket
//! policy has not propagated.
use super::common::{LocalKMSTestEnvironment, create_key_with_specific_id};
use crate::common::{init_logging, local_http_client};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
};
use std::time::Duration;
type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
const DEFAULT_KEY: &str = "kms-anon-default-key";
const BUCKET: &str = "kms-anon-enforcement";
const PLAIN_OBJECT: &str = "plain.txt";
const ENCRYPTED_OBJECT: &str = "encrypted.txt";
const PAYLOAD: &[u8] = b"kms anonymous enforcement payload";
/// How long a bucket policy change may take to reach the request path.
const POLICY_PROPAGATION: Duration = Duration::from_secs(20);
/// Start a local-KMS server and build the public-bucket fixture.
///
/// The bucket holds a plaintext object (the positive control), an SSE-KMS
/// object, an SSE-KMS default-encryption rule, and a bucket policy opening
/// `GetObject`/`PutObject` to everyone. The enforcement switch defaults to off,
/// so the enforcing case has to set it explicitly.
async fn start_public_sse_kms_bucket(env: &mut LocalKMSTestEnvironment, enforce: bool) -> TestResult {
create_key_with_specific_id(&env.kms_keys_dir, DEFAULT_KEY).await?;
let key_dir = env.kms_keys_dir.clone();
let args = vec![
"--kms-enable",
"--kms-backend",
"local",
"--kms-key-dir",
key_dir.as_str(),
"--kms-default-key-id",
DEFAULT_KEY,
];
let mut envs = vec![("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")];
if enforce {
envs.push(("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "true"));
}
env.base_env.start_rustfs_server_with_env(args, &envs).await?;
env.base_env.create_test_bucket(BUCKET).await?;
let owner = env.base_env.create_s3_client();
owner
.put_object()
.bucket(BUCKET)
.key(PLAIN_OBJECT)
.body(ByteStream::from_static(PAYLOAD))
.send()
.await?;
owner
.put_object()
.bucket(BUCKET)
.key(ENCRYPTED_OBJECT)
.body(ByteStream::from_static(PAYLOAD))
.server_side_encryption(ServerSideEncryption::AwsKms)
.ssekms_key_id(DEFAULT_KEY)
.send()
.await?;
let encryption_config = ServerSideEncryptionConfiguration::builder()
.rules(
ServerSideEncryptionRule::builder()
.apply_server_side_encryption_by_default(
ServerSideEncryptionByDefault::builder()
.sse_algorithm(ServerSideEncryption::AwsKms)
.kms_master_key_id(DEFAULT_KEY)
.build()?,
)
.build(),
)
.build()?;
owner
.put_bucket_encryption()
.bucket(BUCKET)
.server_side_encryption_configuration(encryption_config)
.send()
.await?;
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Sid": "PublicReadWrite",
"Effect": "Allow",
"Principal": "*",
"Action": ["s3:GetObject", "s3:PutObject"],
"Resource": [format!("arn:aws:s3:::{BUCKET}/*")]
}]
})
.to_string();
owner.put_bucket_policy().bucket(BUCKET).policy(&policy).send().await?;
let _ = owner.delete_public_access_block().bucket(BUCKET).send().await;
Ok(())
}
fn object_url(env: &LocalKMSTestEnvironment, key: &str) -> String {
format!("{}/{BUCKET}/{key}", env.base_env.url)
}
async fn anonymous_get(env: &LocalKMSTestEnvironment, key: &str) -> Result<reqwest::Response, reqwest::Error> {
local_http_client().get(object_url(env, key)).send().await
}
async fn anonymous_put(env: &LocalKMSTestEnvironment, key: &str) -> Result<reqwest::Response, reqwest::Error> {
local_http_client().put(object_url(env, key)).body(PAYLOAD).send().await
}
/// Retry the plaintext read until the public bucket policy is live.
async fn wait_for_public_read(env: &LocalKMSTestEnvironment) -> TestResult {
let deadline = tokio::time::Instant::now() + POLICY_PROPAGATION;
loop {
let status = anonymous_get(env, PLAIN_OBJECT).await?.status();
if status.as_u16() == 200 {
return Ok(());
}
if tokio::time::Instant::now() >= deadline {
return Err(format!("positive control never became readable: anonymous GET {PLAIN_OBJECT} -> {status}").into());
}
tokio::time::sleep(Duration::from_millis(500)).await;
}
}
async fn assert_anonymous_denied(response: reqwest::Response, what: &str) -> TestResult {
let status = response.status().as_u16();
let body = response.text().await?;
assert_eq!(status, 403, "{what} must be denied, got {status}: {body}");
assert!(body.contains("AccessDenied"), "{what} must carry AccessDenied: {body}");
Ok(())
}
/// Enforcement on: a public bucket policy does not exempt anonymous requests
/// from per-key authorization, on either the read or the default-encryption
/// write path.
#[tokio::test(flavor = "multi_thread")]
async fn anonymous_sse_kms_denied_under_enforcement() -> TestResult {
init_logging();
let mut env = LocalKMSTestEnvironment::new().await?;
start_public_sse_kms_bucket(&mut env, true).await?;
wait_for_public_read(&env).await?;
let read = anonymous_get(&env, ENCRYPTED_OBJECT).await?;
assert_anonymous_denied(read, "anonymous GET of an SSE-KMS object").await?;
let write = anonymous_put(&env, "anon-write.txt").await?;
assert_anonymous_denied(write, "anonymous PUT through an SSE-KMS default-encryption rule").await?;
Ok(())
}
/// Enforcement off (the default): bucket policy alone governs anonymous access,
/// and the default-encryption rule still encrypts anonymous writes.
#[tokio::test(flavor = "multi_thread")]
async fn anonymous_sse_kms_governed_by_bucket_policy_without_enforcement() -> TestResult {
init_logging();
let mut env = LocalKMSTestEnvironment::new().await?;
start_public_sse_kms_bucket(&mut env, false).await?;
wait_for_public_read(&env).await?;
let read = anonymous_get(&env, ENCRYPTED_OBJECT).await?;
assert_eq!(read.status().as_u16(), 200, "anonymous GET of a public SSE-KMS object must succeed");
assert_eq!(read.bytes().await?.as_ref(), PAYLOAD, "the object must be served decrypted");
let write = anonymous_put(&env, "anon-write.txt").await?;
assert_eq!(write.status().as_u16(), 200, "anonymous PUT to a public bucket must succeed");
let stored = env
.base_env
.create_s3_client()
.head_object()
.bucket(BUCKET)
.key("anon-write.txt")
.send()
.await?;
assert_eq!(
stored.server_side_encryption(),
Some(&ServerSideEncryption::AwsKms),
"the anonymous write must be encrypted by the bucket default rule"
);
Ok(())
}
-3
View File
@@ -57,9 +57,6 @@ mod copy_object_version_restore_sse_test;
#[cfg(test)]
mod configured_roundtrip_test;
#[cfg(test)]
mod kms_anonymous_enforcement_test;
#[cfg(test)]
mod kms_authorization_negative_matrix_test;
+3
View File
@@ -218,6 +218,9 @@ faster-hex = { workspace = true }
ratelimit = { workspace = true }
aws-smithy-http-client = { workspace = true, default-features = false, features = ["rustls-aws-lc"] }
# High-performance hashing
ahash = { workspace = true, features = ["serde"] }
# Observability and Metrics
metrics = { workspace = true }
@@ -12,6 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use ahash::AHashMap;
use super::{metadata_boundary, object_lock_boundary, runtime_boundary as runtime_sources};
use crate::bucket::lifecycle::bucket_lifecycle_audit::{
LcAuditEvent, LcEventSrc, emit_non_transitioned_expiration_event, emit_transition_complete_event,
@@ -2870,7 +2871,7 @@ fn spawn_transition_transaction_recovery_once(api: Arc<ECStore>) {
struct StaleMultipartUploadCandidate {
path: String,
initiated: OffsetDateTime,
metadata: Option<HashMap<String, String>>,
metadata: Option<AHashMap<String, String>>,
}
fn parse_stale_uploads_duration(env_key: &str, default: StdDuration) -> StdDuration {
@@ -2915,9 +2916,9 @@ async fn stale_upload_current_size(set: &Arc<SetDisks>, metadata: &HashMap<Strin
stale_upload_current_size_with_opts(set, metadata, upload_dir, false).await
}
async fn stale_upload_current_size_with_opts(
async fn stale_upload_current_size_with_opts<S: std::hash::BuildHasher>(
set: &Arc<SetDisks>,
metadata: &HashMap<String, String>,
metadata: &HashMap<String, String, S>,
upload_dir: &str,
no_lock: bool,
) -> Option<usize> {
@@ -2950,9 +2951,9 @@ async fn stale_upload_current_size_with_opts(
)
}
async fn stale_upload_lifecycle_due(
async fn stale_upload_lifecycle_due<S: std::hash::BuildHasher>(
set: &Arc<SetDisks>,
metadata: &HashMap<String, String>,
metadata: &HashMap<String, String, S>,
initiated: OffsetDateTime,
upload_dir: &str,
no_lock: bool,
@@ -2978,7 +2979,7 @@ async fn stale_upload_lifecycle_due(
.unwrap_or_default(),
is_latest: true,
delete_marker: false,
user_defined: metadata.clone(),
user_defined: metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect(),
..Default::default()
};
+16 -131
View File
@@ -27,7 +27,7 @@ use rustfs_protos::{
ConnectionEvictionLogLevel, evict_failed_connection_with_log_level, models::PingBodyBuilder,
proto_gen::node_service::node_service_client::NodeServiceClient,
};
use std::{sync::OnceLock, time::Duration};
use std::time::Duration;
use tokio::time::timeout;
use tonic::Request;
use tonic::service::interceptor::InterceptedService;
@@ -44,35 +44,11 @@ pub struct RemoteClient {
}
impl RemoteClient {
const ONLINE_CHECK_RESOURCE: &'static str = "health-lock-online";
pub fn new(endpoint: String) -> Self {
Self { addr: endpoint }
}
fn ping_body() -> Bytes {
static BODY: OnceLock<Bytes> = OnceLock::new();
BODY.get_or_init(|| {
let mut fbb = flatbuffers::FlatBufferBuilder::new();
let payload = fbb.create_vector(b"health-check");
let mut builder = PingBodyBuilder::new(&mut fbb);
builder.add_payload(payload);
let root = builder.finish();
fbb.finish(root, None);
Bytes::copy_from_slice(fbb.finished_data())
})
.clone()
}
fn build_ping_request() -> PingRequest {
PingRequest {
version: 1,
body: Self::ping_body(),
}
}
#[cfg(test)]
fn build_fresh_ping_request_for_test() -> PingRequest {
let mut fbb = flatbuffers::FlatBufferBuilder::new();
let payload = fbb.create_vector(b"health-check");
let mut builder = PingBodyBuilder::new(&mut fbb);
@@ -188,16 +164,6 @@ impl RemoteClient {
)
}
fn online_check_timeout() -> Duration {
Duration::from_millis(
rustfs_utils::get_env_u64(
rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS,
rustfs_config::DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS,
)
.max(1),
)
}
async fn execute_rpc<T, F>(&self, op: &'static str, resource_summary: &str, future: F) -> std::result::Result<T, LockError>
where
F: std::future::Future<Output = std::result::Result<T, tonic::Status>>,
@@ -581,37 +547,24 @@ impl LockClient for RemoteClient {
}
async fn is_online(&self) -> bool {
let online_timeout = Self::online_check_timeout();
match timeout(online_timeout, async {
let mut client = self.get_client().await?;
let ping_req = Request::new(Self::build_ping_request());
self.execute_rpc("ping", Self::ONLINE_CHECK_RESOURCE, client.ping(ping_req))
.await?;
Ok::<(), LockError>(())
})
.await
{
Ok(Ok(())) => {
debug!(addr = %self.addr, timeout_ms = online_timeout.as_millis(), "remote lock client is online");
// Use Ping interface to test if remote service is online
let mut client = match self.get_client().await {
Ok(client) => client,
Err(_) => {
info!("remote client {} connection failed", self.addr);
return false;
}
};
let ping_req = Request::new(Self::build_ping_request());
match client.ping(ping_req).await {
Ok(_) => {
info!("remote client {} is online", self.addr);
true
}
Ok(Err(err)) => {
debug!(
addr = %self.addr,
timeout_ms = online_timeout.as_millis(),
error = %err,
"remote lock client online check failed"
);
false
}
Err(_) => {
let reason = format!("online check timed out after {:?}", online_timeout);
warn!(
addr = %self.addr,
timeout_ms = online_timeout.as_millis(),
"remote lock client online check timed out"
);
self.evict_connection("ping", &reason, Self::ONLINE_CHECK_RESOURCE).await;
info!("remote client {} ping failed", self.addr);
false
}
}
@@ -698,15 +651,6 @@ mod tests {
);
}
#[test]
fn cached_ping_request_matches_fresh_flatbuffer_payload() {
let cached = RemoteClient::build_ping_request();
let fresh = RemoteClient::build_fresh_ping_request_for_test();
assert_eq!(cached.version, fresh.version);
assert_eq!(cached.body, fresh.body);
}
#[tokio::test]
#[serial_test::serial]
async fn test_remote_client_acquire_lock_uses_rpc_timeout_and_evicts_connection() {
@@ -835,48 +779,6 @@ mod tests {
accept_task.abort();
}
#[tokio::test]
#[serial_test::serial]
async fn test_remote_client_is_online_uses_health_timeout_and_evicts_connection() {
ensure_test_rpc_secret();
let Some((addr, accept_task)) = spawn_hanging_listener().await else {
return;
};
cache_lazy_channel(&addr).await;
assert!(runtime_sources::test_node_channel_is_cached(&addr).await);
temp_env::async_with_vars(
[
(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("50")),
(rustfs_config::ENV_OBJECT_LOCK_RPC_TIMEOUT_MS, Some("1000")),
],
async {
let client = RemoteClient::new(addr.clone());
let started_at = tokio::time::Instant::now();
let online = client.is_online().await;
let elapsed = started_at.elapsed();
assert!(!online, "hanging remote lock peer must not be reported online");
assert!(
elapsed >= Duration::from_millis(40),
"remote online check should honor configured health timeout, got {elapsed:?}"
);
assert!(
elapsed < Duration::from_secs(1),
"health timeout should keep readiness probes bounded, got {elapsed:?}"
);
assert!(
!runtime_sources::test_node_channel_is_cached(&addr).await,
"online-check timeout should evict cached connection"
);
},
)
.await;
accept_task.abort();
}
#[tokio::test]
#[serial_test::serial]
async fn test_remote_client_refresh_tonic_error_evicts_connection() {
@@ -1004,21 +906,4 @@ mod tests {
assert_eq!(RemoteClient::rpc_timeout(), Duration::from_millis(1));
});
}
#[test]
#[serial_test::serial]
fn test_remote_client_online_timeout_honors_configured_deadline() {
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, None::<&str>, || {
assert_eq!(
RemoteClient::online_check_timeout(),
Duration::from_millis(rustfs_config::DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS)
);
});
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("50"), || {
assert_eq!(RemoteClient::online_check_timeout(), Duration::from_millis(50));
});
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("0"), || {
assert_eq!(RemoteClient::online_check_timeout(), Duration::from_millis(1));
});
}
}
+12 -60
View File
@@ -2793,13 +2793,12 @@ where
#[cfg(test)]
mod tests {
use super::{
SERVER_CONFIG_LOCK, ServerConfigSnapshot, apply_dynamic_config_for_sub_sys_with, build_scalar_config_object,
config_task_join_error, configs_semantically_equal, decode_server_config_blob, encode_server_config_blob,
heal_config_descriptor, is_standard_object_server_config, lookup_configs, new_and_save_server_config, read_config,
read_config_no_lock_preserve_empty_with_metadata, read_config_preserve_empty, read_config_with_metadata,
read_config_without_migrate, read_server_config_snapshot, save_server_config, save_server_config_snapshot,
save_server_config_snapshot_with_generation, server_config_transaction_lock_path, should_warn_ignored_scalar_section,
storage_class_kvs_mut,
SERVER_CONFIG_LOCK, ServerConfigSnapshot, apply_dynamic_config_for_sub_sys_with, config_task_join_error,
configs_semantically_equal, decode_server_config_blob, encode_server_config_blob, is_standard_object_server_config,
lookup_configs, new_and_save_server_config, read_config, read_config_no_lock_preserve_empty_with_metadata,
read_config_preserve_empty, read_config_with_metadata, read_config_without_migrate, read_server_config_snapshot,
save_server_config, save_server_config_snapshot, save_server_config_snapshot_with_generation,
server_config_transaction_lock_path, should_warn_ignored_scalar_section, storage_class_kvs_mut,
};
use crate::config::{audit, heal, notify, oidc, scanner};
use crate::disk::endpoint::Endpoint;
@@ -3542,31 +3541,6 @@ mod tests {
cfg
}
/// `Config::new()` (and every decode built on it) reads the process-global
/// `rustfs_config::server_config::DEFAULT_KVS` OnceLock at call time, and
/// other tests in this binary register it via `crate::config::init()`
/// mid-run. Equality assertions must therefore normalize both sides with
/// one snapshot taken after both configs exist, never against a later
/// `Config::new()`.
fn default_kvs_snapshot() -> Option<&'static std::collections::HashMap<String, KVS>> {
rustfs_config::server_config::DEFAULT_KVS.get()
}
/// Fills the default sections `cfg` is missing from an explicit
/// [`default_kvs_snapshot`], mirroring `Config::set_defaults`.
fn filled_with_default_kvs(mut cfg: Config, snapshot: Option<&std::collections::HashMap<String, KVS>>) -> Config {
if let Some(defaults) = snapshot {
for (sub_sys, kvs) in defaults {
cfg.0
.entry(sub_sys.clone())
.or_default()
.entry(DEFAULT_DELIMITER.to_string())
.or_insert_with(|| kvs.clone());
}
}
cfg
}
#[test]
fn test_external_scanner_config_decodes_with_defaults() {
let cfg =
@@ -3656,9 +3630,7 @@ mod tests {
}"#;
let cfg = decode_server_config_blob(seed).expect("root heal null should mean no persisted override");
// The heal section may hold registered defaults, so assert on the
// semantic diff instead of the section's presence.
assert!(build_scalar_config_object(&cfg, heal_config_descriptor()).is_empty());
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
assert!(!is_standard_object_server_config(seed));
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("legacy seed should canonicalize on an authorized save");
@@ -3693,12 +3665,7 @@ mod tests {
let input = format!(r#"{{"version":"33","storageclass":{{"standard":"","rrs":""}},{section}}}"#);
let cfg = decode_server_config_blob(input.as_bytes())
.unwrap_or_else(|err| panic!("legacy scalar section {section} should be ignored, got: {err}"));
let snapshot = default_kvs_snapshot();
assert_eq!(
filled_with_default_kvs(cfg.clone(), snapshot),
filled_with_default_kvs(base.clone(), snapshot),
"ignored section {section} must contribute no overrides"
);
assert_eq!(cfg, base, "ignored section {section} must contribute no overrides");
assert!(
!is_standard_object_server_config(input.as_bytes()),
"seed with {section} must not count as standard so a save rewrites it"
@@ -3776,19 +3743,12 @@ mod tests {
fn valid_heal_object_and_kvs_array_shapes_remain_accepted() {
let empty_object = br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":{}}"#;
let cfg = decode_server_config_blob(empty_object).expect("empty heal object should decode as no override");
// The heal section may hold registered defaults, so assert on the
// semantic diff instead of the section's presence.
assert!(build_scalar_config_object(&cfg, heal_config_descriptor()).is_empty());
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
let kvs_array =
br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":[{"key":"bitrot_cycle","value":"off"}]}"#;
let cfg = decode_server_config_blob(kvs_array).expect("heal KVS array should decode");
assert_eq!(
build_scalar_config_object(&cfg, heal_config_descriptor())
.get(HEAL_BITROT_CYCLE)
.and_then(Value::as_str),
Some("off")
);
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_some());
}
#[test]
@@ -4940,12 +4900,8 @@ mod tests {
fn test_fallback_returns_default_config_when_recovery_enabled() {
let cfg = fallback_server_config_after_corruption(corrupt_config_error(), "config/config.json", true)
.expect("recovery enabled must fall back to the default config");
let snapshot = default_kvs_snapshot();
assert!(
configs_semantically_equal(
&filled_with_default_kvs(cfg, snapshot),
&filled_with_default_kvs(Config(std::collections::HashMap::new()), snapshot)
),
configs_semantically_equal(&cfg, &Config::new()),
"fallback config should be the default server config"
);
}
@@ -5562,12 +5518,8 @@ mod tests {
.expect("unrecoverable corruption should fall back to the default config");
assert_eq!(store.heal_calls.load(Ordering::SeqCst), 1, "heal should be attempted before falling back");
let snapshot = default_kvs_snapshot();
assert!(
configs_semantically_equal(
&filled_with_default_kvs(cfg, snapshot),
&filled_with_default_kvs(Config(std::collections::HashMap::new()), snapshot)
),
configs_semantically_equal(&cfg, &Config::new()),
"fallback config should be the default server config"
);
}
+3 -31
View File
@@ -96,8 +96,6 @@ const LOG_SUBSYSTEM_POOLS: &str = "pools";
const EVENT_DECOMMISSION_STATE: &str = "decommission_state";
const EVENT_DECOMMISSION_BUCKET: &str = "decommission_bucket";
const EVENT_DECOMMISSION_ENTRY: &str = "decommission_entry";
const POOL_ACTIVATION_FLEET_PROOF_REQUIRED: &str = "pool activation requires a live fleet capability proof";
const POOL_ACTIVATION_FLEET_PROOF_EXPIRED: &str = "pool activation fleet capability proof expired before commit";
const DECOMMISSION_STAGE_MIGRATE_OBJECT: &str = "migrate_object";
const DECOMMISSION_STAGE_CLEANUP_PREFLIGHT: &str = "cleanup_preflight";
const DECOMMISSION_STAGE_SOURCE_CLEANUP: &str = "source_cleanup";
@@ -1834,13 +1832,6 @@ pub(crate) struct PoolRebalanceActivationFence {
}
impl PoolRebalanceActivationFence {
pub(crate) fn set_fleet_proof(
&mut self,
fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
) {
self.fleet_proof = fleet_proof;
}
pub(crate) fn ensure_held(&self) -> Result<()> {
#[cfg(test)]
let forced_lost = self.forced_lost.load(Ordering::Acquire);
@@ -1854,7 +1845,7 @@ impl PoolRebalanceActivationFence {
.as_ref()
.is_some_and(|proof| !crate::services::notification_sys::cross_pool_fence_fleet_proof_matches(proof))
{
return Err(Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED));
return Err(Error::other("pool activation fleet capability proof expired before commit"));
}
Ok(())
@@ -1910,17 +1901,7 @@ pub(crate) async fn acquire_pool_activation_fleet_proof(
}
crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof()
.map(Some)
.ok_or_else(|| Error::other(POOL_ACTIVATION_FLEET_PROOF_REQUIRED))
}
pub(crate) fn is_pool_activation_fleet_proof_error(err: &Error) -> bool {
// Save-stage helpers add context by formatting the original error, so the
// marker may be nested in the display string. Restrict matching to the
// `Error::other` I/O shape used by this activation path.
matches!(err, Error::Io(io_error) if io_error.kind() == std::io::ErrorKind::Other && {
let message = io_error.to_string();
message.contains(POOL_ACTIVATION_FLEET_PROOF_REQUIRED) || message.contains(POOL_ACTIVATION_FLEET_PROOF_EXPIRED)
})
.ok_or_else(|| Error::other("pool activation requires a live fleet capability proof"))
}
#[cfg(test)]
@@ -5815,7 +5796,7 @@ fn decommission_remote_tiered_opts(
versioned: version_id.is_some(),
version_id,
mod_time: version.mod_time,
user_defined: version.metadata.clone(),
user_defined: version.metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect(),
src_pool_idx,
data_movement: true,
incl_free_versions: version.tier_free_version(),
@@ -10847,15 +10828,6 @@ mod tests {
use crate::bucket::replication::{ReplicationState, ReplicationStatusType};
use serde::Serialize;
#[test]
fn pool_activation_fleet_proof_error_classifier_matches_only_retryable_proof_failures() {
assert!(is_pool_activation_fleet_proof_error(&Error::other(POOL_ACTIVATION_FLEET_PROOF_REQUIRED)));
assert!(is_pool_activation_fleet_proof_error(&Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED)));
let wrapped = format!("rebalance meta save failed during start_rebalance: {POOL_ACTIVATION_FLEET_PROOF_EXPIRED}");
assert!(is_pool_activation_fleet_proof_error(&Error::other(wrapped)));
assert!(!is_pool_activation_fleet_proof_error(&Error::ConfigNotFound));
}
#[tokio::test]
#[serial_test::serial]
async fn decommission_activation_fence_loss_after_durable_save_blocks_publication() {
-5
View File
@@ -157,8 +157,6 @@ pub enum StorageError {
InvalidPartNumber(usize),
#[error("Your proposed upload is smaller than the minimum allowed size. Part {0} size {1} is less than minimum {2}")]
EntityTooSmall(usize, i64, i64),
#[error("multipart upload size {0} exceeds the configured limit {1}")]
EntityTooLarge(u64, u64),
// ── Erasure / Quorum ─────────────────────────────────────────────
#[error("erasure read quorum")]
@@ -556,7 +554,6 @@ impl Clone for StorageError {
StorageError::DecommissionNotStarted => StorageError::DecommissionNotStarted,
StorageError::InvalidPart(a, b, c) => StorageError::InvalidPart(*a, b.clone(), c.clone()),
StorageError::EntityTooSmall(a, b, c) => StorageError::EntityTooSmall(*a, *b, *c),
StorageError::EntityTooLarge(a, b) => StorageError::EntityTooLarge(*a, *b),
StorageError::DoneForNow => StorageError::DoneForNow,
StorageError::DecommissionAlreadyRunning => StorageError::DecommissionAlreadyRunning,
StorageError::RebalanceAlreadyRunning => StorageError::RebalanceAlreadyRunning,
@@ -676,7 +673,6 @@ impl StorageError {
StorageError::InsufficientWriteQuorum(_, _) => StorageErrorCode::InsufficientWriteQuorum,
StorageError::PreconditionFailed => StorageErrorCode::PreconditionFailed,
StorageError::EntityTooSmall(_, _, _) => StorageErrorCode::EntityTooSmall,
StorageError::EntityTooLarge(_, _) => StorageErrorCode::EntityTooLarge,
StorageError::InvalidRangeSpec(_) => StorageErrorCode::InvalidRangeSpec,
StorageError::NotModified => StorageErrorCode::NotModified,
StorageError::InvalidPartNumber(_) => StorageErrorCode::InvalidPartNumber,
@@ -799,7 +795,6 @@ impl StorageError {
StorageErrorCode::EntityTooSmall => {
Some(StorageError::EntityTooSmall(Default::default(), Default::default(), Default::default()))
}
StorageErrorCode::EntityTooLarge => Some(StorageError::EntityTooLarge(Default::default(), Default::default())),
StorageErrorCode::InvalidRangeSpec => Some(StorageError::InvalidRangeSpec(Default::default())),
StorageErrorCode::NotModified => Some(StorageError::NotModified),
StorageErrorCode::InvalidPartNumber => Some(StorageError::InvalidPartNumber(Default::default())),
+1 -1
View File
@@ -64,7 +64,7 @@ pub(crate) const ENCRYPTED_FRAME_LAYOUT_FIXED8K_SUFFIX: &str = "encrypted-frame-
pub(crate) const ENV_RUSTFS_ENCRYPTED_RANGE_SEEK: &str = "RUSTFS_ENCRYPTED_RANGE_SEEK";
pub(crate) const DEFAULT_RUSTFS_ENCRYPTED_RANGE_SEEK: bool = true;
pub(crate) fn has_encrypted_part_layout_marker(metadata: &HashMap<String, String>, suffix: &str, expected: &str) -> bool {
pub(crate) fn has_encrypted_part_layout_marker<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>, suffix: &str, expected: &str) -> bool {
let mut value = None;
for (key, candidate) in metadata {
if !rustfs_utils::http::has_internal_suffix(key, suffix) {
@@ -168,7 +168,7 @@ pub fn to_s3s_etag(etag: &str) -> ETag {
ETag::Strong(etag.to_string())
}
pub fn get_raw_etag(metadata: &HashMap<String, String>) -> String {
pub fn get_raw_etag<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> String {
metadata
.get("etag")
.cloned()
+2 -2
View File
@@ -1008,7 +1008,7 @@ impl ObjectInfo {
successor_mod_time: fi.successor_mod_time,
etag,
inlined,
user_defined: Arc::new(metadata),
user_defined: Arc::new(metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect()),
transitioned_object,
transition_version_state: fi.transition_version_state,
checksum: fi.checksum.clone(),
@@ -1316,7 +1316,7 @@ impl ObjectInfo {
if part > 0
&& let Some(checksums) = self.parts.iter().find(|p| p.number == part).and_then(|p| p.checksums.clone())
{
return Ok((checksums, true));
return Ok((checksums.iter().map(|(k, v)| (k.clone(), v.clone())).collect(), true));
}
if let Some(data) = &self.checksum {
@@ -234,39 +234,6 @@ pub(crate) fn install_cross_pool_fence_fleet_proof_for_test() {
});
}
#[cfg(test)]
pub(crate) struct CrossPoolFenceFleetProofGuard {
previous_proof: Option<FleetCapabilityProof>,
previous_topology_conflict: bool,
}
#[cfg(test)]
impl Drop for CrossPoolFenceFleetProofGuard {
fn drop(&mut self) {
let mut state = cross_pool_fence_fleet_proof_slot()
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
state.proof = self.previous_proof.take();
state.topology_conflict = self.previous_topology_conflict;
}
}
/// Temporarily revoke the test proof so activation paths can exercise their
/// fail-closed behavior without changing the process-wide topology binding.
#[cfg(test)]
pub(crate) fn without_cross_pool_fence_fleet_proof_for_test() -> CrossPoolFenceFleetProofGuard {
let mut state = cross_pool_fence_fleet_proof_slot()
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let guard = CrossPoolFenceFleetProofGuard {
previous_proof: state.proof.clone(),
previous_topology_conflict: state.topology_conflict,
};
state.proof = None;
state.topology_conflict = true;
guard
}
#[cfg(any(test, feature = "test-util"))]
pub fn rotate_cross_pool_fence_fleet_proof_for_test() -> bool {
let mut state = cross_pool_fence_fleet_proof_slot()
@@ -570,13 +570,10 @@ impl ECStore {
where
S: EcstoreObjectIO + StorageNamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
{
// Lock order: pool_meta_save_gate -> pool.bin -> rebalance.bin.
let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
pool_meta_guard.ensure_write_safe("rebalance worker activation")?;
// Classify the durable rebalance record while holding both namespace
// fences. A terminal record is a no-op and must not depend on the
// notification subsystem having published a fleet proof yet.
let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?;
let activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), fleet_proof).await?;
let pool_meta = self
.load_runtime_pool_meta_under_activation_fence(&mut pool_meta_guard, &activation_fence, "rebalance worker activation")
.await?;
@@ -600,17 +597,10 @@ impl ECStore {
}
activation_fence.ensure_held()?;
if !crate::services::rebalance::rebalance_requires_worker_activation(&persisted) {
if !is_rebalance_conflicting_with_decommission(&persisted) {
return Ok(RebalanceWorkerActivationFence::NotStartedTerminal);
}
// Active worker admission still requires the fail-closed fleet proof.
// Attach it immediately before the final fence validation so expiry or
// topology changes are checked again at every later commit boundary.
let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
activation_fence.set_fleet_proof(fleet_proof);
activation_fence.ensure_held()?;
Ok(RebalanceWorkerActivationFence::Ready(Box::new(activation_fence)))
}
@@ -1486,64 +1476,6 @@ mod tests {
assert_activation_locks_released(&store).await;
}
#[tokio::test]
#[serial_test::serial]
async fn rebalance_worker_skips_terminal_metadata_without_fleet_proof() {
let rebalance_id = "terminal-metadata-without-proof";
let completed = RebalanceMeta {
id: rebalance_id.to_string(),
pool_stats: vec![RebalanceStats {
participating: true,
info: RebalanceInfo {
status: RebalStatus::Completed,
..Default::default()
},
..Default::default()
}],
..Default::default()
};
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(Some(completed)).await;
let _proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
let activation = store
.fence_rebalance_worker_activation(store.pools[0].clone(), rebalance_id)
.await
.expect("terminal metadata should not require a fleet proof");
assert!(matches!(activation, RebalanceWorkerActivationFence::NotStartedTerminal));
}
#[tokio::test]
#[serial_test::serial]
async fn rebalance_worker_still_requires_fleet_proof_for_active_metadata() {
let rebalance_id = "active-metadata-without-proof";
let active = RebalanceMeta {
id: rebalance_id.to_string(),
pool_stats: vec![RebalanceStats {
participating: true,
info: RebalanceInfo {
status: RebalStatus::Started,
..Default::default()
},
..Default::default()
}],
..Default::default()
};
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(Some(active)).await;
let _proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
let err = match store
.fence_rebalance_worker_activation(store.pools[0].clone(), rebalance_id)
.await
{
Ok(_) => panic!("active metadata must not be admitted without a fleet proof"),
Err(err) => err,
};
assert!(
err.to_string()
.contains("pool activation requires a live fleet capability proof")
);
}
#[tokio::test]
#[serial_test::serial]
async fn rebalance_activation_adopts_commit_after_post_save_fence_loss() {
@@ -214,14 +214,6 @@ pub(super) fn is_rebalance_in_progress(meta: &RebalanceMeta) -> bool {
meta.pool_stats.iter().any(is_rebalance_pool_active)
}
/// Persisted rebalance metadata requires worker activation only while it has
/// not reached a durable terminal marker and at least one pool is still marked
/// active. Merely finding `rebalance.bin` is not evidence that admission is
/// required: terminal metadata is retained for status reporting.
pub(crate) fn rebalance_requires_worker_activation(meta: &RebalanceMeta) -> bool {
meta.stopped_at.is_none() && is_rebalance_in_progress(meta)
}
pub(crate) fn is_rebalance_conflicting_with_decommission(meta: &RebalanceMeta) -> bool {
is_rebalance_in_progress(meta)
}
@@ -57,7 +57,7 @@ fn rebalance_remote_tiered_opts(
versioned: version_id.is_some(),
version_id,
mod_time: version.mod_time,
user_defined: version.metadata.clone(),
user_defined: version.metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect(),
src_pool_idx,
data_movement: true,
include_part_checksums: true,
+1 -1
View File
@@ -49,8 +49,8 @@ mod worker;
#[cfg(feature = "test-util")]
pub use entry::test_util::PausedRebalanceEntryTestFixture;
pub(crate) use meta::is_rebalance_conflicting_with_decommission;
pub use meta::{decode_rebalance_stop_propagation_record, encode_rebalance_stop_propagation_record};
pub(crate) use meta::{is_rebalance_conflicting_with_decommission, rebalance_requires_worker_activation};
pub use types::{
DiskStat, RebalSaveOpt, RebalStatus, RebalanceCleanupWarningEntry, RebalanceCleanupWarnings, RebalanceInfo, RebalanceMeta,
RebalanceStats, RebalanceStopPropagationRecord,
@@ -22,11 +22,11 @@ use super::meta::{
is_rebalance_in_progress, is_rebalance_meta_replaceable_for_new_id, is_rebalance_stopped_terminal_event,
mark_rebalance_bucket_done, merge_rebalance_bucket_lists, merge_rebalance_meta, next_rebal_bucket_from_stat,
percent_free_ratio, rebalance_goal_reached, rebalance_meta_load_no_data_error, rebalance_meta_load_unknown_format_error,
rebalance_meta_load_unknown_version_error, rebalance_requires_worker_activation, record_rebalance_cleanup_warning_in_meta,
remove_rebalanced_buckets_from_queue, resolve_next_rebalance_bucket, resolve_rebalance_participants,
should_accept_rebalance_stats_update, should_ignore_rebalance_data_usage_cache, should_pool_participate,
should_preserve_rebalance_stopped_state, should_skip_start_rebalance, stop_rebalance_meta_snapshot, stop_rebalance_state,
take_bucket_from_rebalance_queue, validate_init_rebalance_state, validate_start_rebalance_state,
rebalance_meta_load_unknown_version_error, record_rebalance_cleanup_warning_in_meta, remove_rebalanced_buckets_from_queue,
resolve_next_rebalance_bucket, resolve_rebalance_participants, should_accept_rebalance_stats_update,
should_ignore_rebalance_data_usage_cache, should_pool_participate, should_preserve_rebalance_stopped_state,
should_skip_start_rebalance, stop_rebalance_meta_snapshot, stop_rebalance_state, take_bucket_from_rebalance_queue,
validate_init_rebalance_state, validate_start_rebalance_state,
};
use super::migration::{
MigrationBackend, MigrationVersionResult, migrate_entry_version, migrate_entry_version_with_retry_wait,
@@ -3386,52 +3386,6 @@ fn test_is_rebalance_in_progress_only_started_participants() {
assert!(is_rebalance_in_progress(&meta));
}
#[test]
fn test_rebalance_requires_worker_activation_only_for_active_non_stopped_metadata() {
let now = OffsetDateTime::now_utc();
let active = RebalanceMeta {
pool_stats: vec![RebalanceStats {
participating: true,
info: RebalanceInfo {
status: RebalStatus::Started,
..Default::default()
},
..Default::default()
}],
..Default::default()
};
let stopped_active = RebalanceMeta {
stopped_at: Some(now),
pool_stats: active.pool_stats.clone(),
..Default::default()
};
assert!(rebalance_requires_worker_activation(&active));
for status in [
RebalStatus::Completed,
RebalStatus::Stopped,
RebalStatus::Failed,
RebalStatus::None,
] {
let terminal = RebalanceMeta {
pool_stats: vec![RebalanceStats {
participating: true,
info: RebalanceInfo {
status,
..Default::default()
},
..Default::default()
}],
..Default::default()
};
assert!(
!rebalance_requires_worker_activation(&terminal),
"terminal status {status:?} must not resume"
);
}
assert!(!rebalance_requires_worker_activation(&stopped_active));
}
#[test]
fn test_is_rebalance_conflicting_with_decommission_true_when_in_progress() {
let now = OffsetDateTime::now_utc();
+1 -1
View File
@@ -1454,7 +1454,7 @@ fn tier_backend_identity(config: &TierConfig) -> io::Result<TierDestinationId> {
encode_tier_backend_identity(tier_type, endpoint, bucket, prefix, region, routing_account)
}
pub(crate) fn tier_destination_id_from_metadata(metadata: &HashMap<String, String>) -> io::Result<Option<TierDestinationId>> {
pub(crate) fn tier_destination_id_from_metadata<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> io::Result<Option<TierDestinationId>> {
let Some(encoded) = rustfs_utils::http::metadata_compat::get_consistent_str(
metadata,
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID,
+2 -2
View File
@@ -609,7 +609,7 @@ impl SetDisks {
|| Self::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX)
}
fn update_hash_quorum_metadata_map(hasher: &mut Sha256, entries: &HashMap<String, String>) {
fn update_hash_quorum_metadata_map<S: std::hash::BuildHasher>(hasher: &mut Sha256, entries: &HashMap<String, String, S>) {
let mut entries = entries
.iter()
.filter(|(name, _)| !Self::is_replication_quorum_metadata_key(name))
@@ -635,7 +635,7 @@ impl SetDisks {
/// so the dual internal prefixes carrying the same mapping share one
/// identity, while a genuine disagreement between disks still changes the
/// hash and surfaces as a quorum difference.
fn update_hash_target_delete_marker_versions(hasher: &mut Sha256, metadata: &HashMap<String, String>) {
fn update_hash_target_delete_marker_versions<S: std::hash::BuildHasher>(hasher: &mut Sha256, metadata: &HashMap<String, String, S>) {
let (versions, corrupt) = http::target_delete_marker_versions(metadata);
hasher.update([u8::from(corrupt)]);
let mut versions = versions.iter().collect::<Vec<_>>();
+5 -5
View File
@@ -190,7 +190,7 @@ use tracing::error;
use tracing::{Instrument, debug, info, warn};
use uuid::Uuid;
pub(super) fn restore_operation_id_from_metadata(metadata: &HashMap<String, String>) -> Result<Option<Uuid>> {
pub(super) fn restore_operation_id_from_metadata<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> Result<Option<Uuid>> {
let Some(value) = rustfs_utils::http::metadata_compat::get_consistent_str(metadata, SUFFIX_RESTORE_OPERATION_ID) else {
if rustfs_utils::http::metadata_compat::contains_key_str(metadata, SUFFIX_RESTORE_OPERATION_ID) {
return Err(Error::other("invalid restore operation id metadata".to_string()));
@@ -204,14 +204,14 @@ pub(super) fn restore_operation_id_from_metadata(metadata: &HashMap<String, Stri
Ok(Some(id))
}
pub(super) fn require_restore_operation_id(metadata: &HashMap<String, String>, expected: Uuid) -> Result<()> {
pub(super) fn require_restore_operation_id<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>, expected: Uuid) -> Result<()> {
match restore_operation_id_from_metadata(metadata)? {
Some(actual) if actual == expected => Ok(()),
_ => Err(Error::other("restore operation id changed before copy-back".to_string())),
}
}
pub(super) fn restore_commit_operation_id_from_metadata(metadata: &HashMap<String, String>) -> Result<Option<Uuid>> {
pub(super) fn restore_commit_operation_id_from_metadata<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> Result<Option<Uuid>> {
if !metadata.contains_key(X_AMZ_RESTORE.as_str()) {
return Ok(None);
}
@@ -466,13 +466,13 @@ fn release_materialized_read_lock(bucket: &str, object: &str, read_lock_guard: O
drop(read_lock_guard);
}
pub(crate) fn strip_internal_multipart_metadata(metadata: &mut HashMap<String, String>) {
pub(crate) fn strip_internal_multipart_metadata<S: std::hash::BuildHasher>(metadata: &mut HashMap<String, String, S>) {
metadata.remove(RUSTFS_MULTIPART_BUCKET_KEY);
metadata.remove(RUSTFS_MULTIPART_OBJECT_KEY);
rustfs_utils::http::metadata_compat::remove_str(metadata, SUFFIX_BUCKET_INCARNATION_ID);
}
fn should_persist_encryption_original_size(metadata: &HashMap<String, String>) -> bool {
fn should_persist_encryption_original_size<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> bool {
metadata.keys().any(|key| is_object_encryption_marker(key))
}
+42 -293
View File
@@ -84,13 +84,11 @@ use rustfs_rio::TryGetIndex;
use rustfs_utils::http::SSEC_ALGORITHM_HEADER;
#[cfg(test)]
use rustfs_utils::http::SUFFIX_COMPRESSION;
use rustfs_utils::http::{SUFFIX_MAX_TOTAL_OBJECT_SIZE, get_consistent_str};
use std::future::Future;
#[cfg(test)]
use std::sync::atomic::AtomicBool;
#[cfg(any(test, feature = "test-util"))]
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Mutex, OnceLock};
#[cfg(any(test, feature = "test-util"))]
use std::time::Duration;
#[cfg(test)]
@@ -99,83 +97,6 @@ use tokio::task::JoinSet;
const MULTIPART_LIST_IO_CONCURRENCY: usize = 16;
static CAPPED_MULTIPART_STAGING: OnceLock<Mutex<HashMap<String, Arc<tokio::sync::Semaphore>>>> = OnceLock::new();
struct CappedMultipartStagingGuard {
upload_id_path: String,
permit: Option<tokio::sync::OwnedSemaphorePermit>,
}
impl Drop for CappedMultipartStagingGuard {
fn drop(&mut self) {
// Release the permit before checking the Arc count so a concurrent
// Abort/Complete cleanup can remove the now-unused map entry.
self.permit.take();
remove_capped_multipart_staging_semaphore(&self.upload_id_path);
}
}
fn capped_multipart_staging_semaphore(upload_id_path: &str) -> Arc<tokio::sync::Semaphore> {
CAPPED_MULTIPART_STAGING
.get_or_init(|| Mutex::new(HashMap::new()))
.lock()
.expect("capped multipart staging semaphore map should not be poisoned")
.entry(upload_id_path.to_owned())
.or_insert_with(|| Arc::new(tokio::sync::Semaphore::new(1)))
.clone()
}
fn remove_capped_multipart_staging_semaphore(upload_id_path: &str) {
if let Some(map) = CAPPED_MULTIPART_STAGING.get() {
let mut map = map
.lock()
.expect("capped multipart staging semaphore map should not be poisoned");
let removable = map
.get(upload_id_path)
.is_some_and(|semaphore| Arc::strong_count(semaphore) == 1);
if removable {
map.remove(upload_id_path);
}
}
}
fn multipart_size_limit_from_metadata(metadata: &HashMap<String, String>) -> Result<Option<u64>> {
if !contains_key_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE) {
return Ok(None);
}
let Some(value) = get_consistent_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE) else {
return Err(Error::InvalidArgument(
"multipart upload".to_string(),
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
"missing or conflicting internal size limit".to_string(),
));
};
let limit = value.parse::<u64>().map_err(|_| {
Error::InvalidArgument(
"multipart upload".to_string(),
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
"invalid internal size limit".to_string(),
)
})?;
Ok(Some(limit))
}
fn admitted_multipart_size(current: u64, candidate: u64, limit: u64) -> Result<u64> {
let total = current.checked_add(candidate).ok_or_else(|| {
Error::InvalidArgument(
"multipart upload".to_string(),
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
"logical size overflow".to_string(),
)
})?;
if total > limit {
return Err(Error::EntityTooLarge(total, limit));
}
Ok(total)
}
pub(crate) struct StaleMultipartCleanupGuard {
file_info: FileInfo,
upload_path: String,
@@ -194,13 +115,8 @@ impl StaleMultipartCleanupGuard {
pub(crate) async fn delete(self, set: &SetDisks) -> Result<()> {
fence_commit_on_lock_loss(Some(&self.lock_guard), "stale_multipart_cleanup", &self.upload_path)?;
let result = set
.delete_all_with_quorum(RUSTFS_META_MULTIPART_BUCKET, &self.upload_path, self.write_quorum)
.await;
if result.is_ok() {
remove_capped_multipart_staging_semaphore(&self.upload_path);
}
result
set.delete_all_with_quorum(RUSTFS_META_MULTIPART_BUCKET, &self.upload_path, self.write_quorum)
.await
}
}
@@ -449,7 +365,7 @@ fn fence_commit_on_lock_loss(guard: Option<&ObjectLockDiagGuard>, mode: &'static
Ok(())
}
fn multipart_bucket_incarnation_id(metadata: &HashMap<String, String>) -> Result<Option<Uuid>> {
fn multipart_bucket_incarnation_id<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> Result<Option<Uuid>> {
let Some(value) = rustfs_utils::http::metadata_compat::get_consistent_str(metadata, SUFFIX_BUCKET_INCARNATION_ID) else {
if rustfs_utils::http::metadata_compat::contains_key_str(metadata, SUFFIX_BUCKET_INCARNATION_ID) {
return Err(Error::other("invalid multipart bucket incarnation metadata"));
@@ -463,12 +379,12 @@ fn multipart_bucket_incarnation_id(metadata: &HashMap<String, String>) -> Result
Ok(Some(incarnation))
}
fn multipart_bucket_incarnation_matches(metadata: &HashMap<String, String>, expected: Uuid) -> bool {
fn multipart_bucket_incarnation_matches<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>, expected: Uuid) -> bool {
matches!(multipart_bucket_incarnation_id(metadata), Ok(Some(actual)) if actual == expected)
}
fn validate_multipart_bucket_incarnation(
metadata: &HashMap<String, String>,
fn validate_multipart_bucket_incarnation<S: std::hash::BuildHasher>(
metadata: &HashMap<String, String, S>,
bucket: &str,
object: &str,
upload_id: &str,
@@ -719,61 +635,6 @@ async fn multipart_upload_paths_on_disk(disk: DiskStore, bucket: &str) -> disk::
}
impl SetDisks {
async fn current_multipart_logical_size(
&self,
bucket: &str,
object: &str,
upload_id: &str,
upload_id_path: &str,
fi: &FileInfo,
replacing_part: usize,
) -> Result<u64> {
let online_disks = self.get_disks_internal().await;
let read_quorum = fi.read_quorum(self.default_read_quorum());
let part_path = format!(
"{}{}",
path_join_buf(&[
upload_id_path,
fi.data_dir.map(|v| v.to_string()).unwrap_or_default().as_str(),
]),
SLASH_SEPARATOR
);
let part_numbers = match Self::list_parts(&online_disks, &part_path, read_quorum).await {
Ok(parts) => parts,
Err(DiskError::FileNotFound) => return Ok(0),
Err(err) => return Err(to_object_err(err.into(), vec![bucket, object, upload_id])),
};
if part_numbers.is_empty() {
return Ok(0);
}
let part_meta_paths = part_numbers
.iter()
.map(|number| format!("{part_path}part.{number}.meta"))
.collect::<Vec<_>>();
let existing_parts =
Self::read_parts(&online_disks, RUSTFS_META_MULTIPART_BUCKET, &part_meta_paths, &part_numbers, read_quorum)
.await
.map_err(|err| to_object_err(err.into(), vec![bucket, object, upload_id]))?;
existing_parts.into_iter().try_fold(0_u64, |total, part| {
if part.error.is_some() || part.number == replacing_part {
return if part.error.is_some() {
Err(Error::PartMissingOrCorrupt)
} else {
Ok(total)
};
}
let part_size = u64::try_from(part.actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
total.checked_add(part_size).ok_or_else(|| {
Error::InvalidArgument(
"multipart upload".to_string(),
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
"logical size overflow".to_string(),
)
})
})
}
async fn discover_multipart_upload_paths(
&self,
orig_bucket: &str,
@@ -1269,18 +1130,9 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
crate::hp_guard!("SetDisks::put_object_part");
let upload_id_path = Self::get_multipart_upload_dir(bucket, object, upload_id, opts.data_movement);
let (fi, _) = match self
let (fi, _) = self
.check_upload_id_exists_with_opts(bucket, object, upload_id, true, opts)
.await
{
Ok(value) => value,
Err(err @ Error::InvalidUploadID(..)) => {
remove_capped_multipart_staging_semaphore(&upload_id_path);
return Err(err);
}
Err(err) => return Err(err),
};
let multipart_size_limit = multipart_size_limit_from_metadata(&fi.metadata)?;
.await?;
ensure_data_movement_upload_access(&fi, bucket, object, upload_id, opts)?;
ensure_multipart_bucket_incarnation(&self.ctx, &fi, bucket, object, upload_id, opts.expected_bucket_incarnation_id)
.await?;
@@ -1313,44 +1165,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
let part_suffix = format!("part.{part_id}");
let tmp_part = format!("{}x{}", Uuid::new_v4(), OffsetDateTime::now_utc().unix_timestamp());
let tmp_part_path = Arc::new(format!("{tmp_part}/{part_suffix}"));
let part_path = format!("{}/{}/{}", upload_id_path, fi.data_dir.unwrap_or_default(), part_suffix);
let part_lock_path = format!("{upload_id_path}/{part_suffix}");
// Keep at most one capped part staging locally per upload. The
// distributed lock below is held only for the durable admission check;
// it is reacquired for the short final rename, so Complete/Abort are
// not blocked behind a slow body upload.
let _capped_staging_guard = if multipart_size_limit.is_some() {
Some(CappedMultipartStagingGuard {
upload_id_path: upload_id_path.clone(),
permit: Some(
capped_multipart_staging_semaphore(&upload_id_path)
.acquire_owned()
.await
.map_err(|_| Error::other("capped multipart staging semaphore closed"))?,
),
})
} else {
None
};
if let Some(limit) = multipart_size_limit {
let admission_guard = self
.acquire_write_lock_diag("put_object_part_admission", RUSTFS_META_MULTIPART_BUCKET, &upload_id_path)
.await?;
let declared_size = if data.size() >= 0 {
u64::try_from(data.size()).map_err(|_| Error::PartMissingOrCorrupt)?
} else if data.actual_size() >= 0 {
u64::try_from(data.actual_size()).map_err(|_| Error::PartMissingOrCorrupt)?
} else {
return Err(Error::PartMissingOrCorrupt);
};
let current_size = self
.current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &fi, part_id)
.await?;
admitted_multipart_size(current_size, declared_size, limit)?;
drop(admission_guard);
}
let result: Result<PartInfo> = async {
let erasure =
@@ -1521,7 +1335,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
mod_time: Some(OffsetDateTime::now_utc()),
actual_size,
index: index_op,
checksums: if checksums.is_empty() { None } else { Some(checksums) },
checksums: if checksums.is_empty() { None } else { Some(checksums.iter().map(|(k, v)| (k.clone(), v.clone())).collect()) },
..Default::default()
};
@@ -1551,21 +1365,30 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
.await?;
}
let part_path = format!("{}/{}/{}", upload_id_path, fi.data_dir.unwrap_or_default(), part_suffix);
let part_lock_path = format!("{upload_id_path}/{part_suffix}");
#[cfg(test)]
pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockAcquire).await;
// Capped uploads reacquire the upload-wide write lock for the
// final durable check and rename. Uncapped uploads retain the
// concurrent encode path and only serialize the final same-part
// rename; completion/abort use the upload-wide write lock.
let (_upload_commit_guard, _part_commit_guard) = if multipart_size_limit.is_some() {
let upload_guard = self
.acquire_write_lock_diag("put_object_part_commit", RUSTFS_META_MULTIPART_BUCKET, &upload_id_path)
.await?;
let part_guard = self
.acquire_write_lock_diag("put_object_part_commit", RUSTFS_META_MULTIPART_BUCKET, &part_lock_path)
.await?;
(Some(upload_guard), Some(part_guard))
} else if opts.no_lock {
// Serialize only same-part commits (rename_part), not the whole upload.
// Each concurrent stream writes to its own unique temp dir (see
// `tmp_part` above), so the encode/stream phase never conflicts and must
// stay lock-free — holding a lock across it would serialize slow
// re-transmits of the same part and defeat the S3 "last finisher wins"
// semantics. The mixed-generation hazard is confined to rename_part,
// where two temp parts are moved cross-disk onto the SAME final
// part_path: interleaving there can leave shards from two generations,
// each individually bitrot-valid, that only surface as silent corruption
// at read time (backlog#853). A write lock scoped to this part number
// makes each same-part commit atomic across disks, so the last committer
// wins consistently, while different part numbers commit onto disjoint
// part paths and stay concurrent (issue#5961 — an uploadId-wide write
// lock serialized them into 503 lock-acquire timeouts). The shared
// uploadId read lock keeps completion/abort (which take the uploadId
// write lock) from racing any in-flight part commit; a guarded
// completion takes the object lock before the upload lock to preserve
// global ordering.
let (_upload_commit_guard, _part_commit_guard) = if opts.no_lock {
(None, None)
} else {
let upload_guard = self
@@ -1577,16 +1400,8 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
(Some(upload_guard), Some(part_guard))
};
let (commit_fi, _) = self
.check_upload_id_exists_with_opts(bucket, object, upload_id, multipart_size_limit.is_some(), opts)
.check_upload_id_exists_with_opts(bucket, object, upload_id, false, opts)
.await?;
let commit_size_limit = multipart_size_limit_from_metadata(&commit_fi.metadata)?;
if commit_size_limit != multipart_size_limit {
return Err(Error::InvalidArgument(
"multipart upload".to_string(),
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
"size limit metadata changed or is missing".to_string(),
));
}
ensure_data_movement_upload_access(&commit_fi, bucket, object, upload_id, opts)?;
ensure_multipart_bucket_incarnation(
&self.ctx,
@@ -1616,14 +1431,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
}
ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
if let Some(limit) = commit_size_limit {
let current_size = self
.current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &commit_fi, part_id)
.await?;
let candidate_size = u64::try_from(actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
admitted_multipart_size(current_size, candidate_size, limit)?;
}
let _ = self
.rename_part(
&shuffle_disks,
@@ -1716,7 +1523,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
max_parts,
part_number_marker,
user_defined: {
let mut metadata = fi.metadata.clone();
let mut metadata: HashMap<String, String> = fi.metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
strip_internal_multipart_metadata(&mut metadata);
metadata
},
@@ -1975,7 +1782,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
let mod_time = opts.mod_time.unwrap_or_else(OffsetDateTime::now_utc);
for f in parts_metadatas.iter_mut() {
f.metadata = user_defined.clone();
f.metadata = user_defined.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
f.mod_time = Some(mod_time);
f.fresh = true;
}
@@ -2064,7 +1871,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
upload_id: upload_id.to_owned(),
user_defined: {
strip_internal_multipart_metadata(&mut fi.metadata);
fi.metadata.clone()
fi.metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect()
},
..Default::default()
})
@@ -2084,17 +1891,12 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
let upload_id_path = Self::get_multipart_upload_dir(bucket, object, upload_id, opts.data_movement);
let result = self
.delete_all_with_quorum(
RUSTFS_META_MULTIPART_BUCKET,
&upload_id_path,
fi.write_quorum(self.default_write_quorum()),
)
.await;
if result.is_ok() {
remove_capped_multipart_staging_semaphore(&upload_id_path);
}
result
self.delete_all_with_quorum(
RUSTFS_META_MULTIPART_BUCKET,
&upload_id_path,
fi.write_quorum(self.default_write_quorum()),
)
.await
}
// complete_multipart_upload finished
#[tracing::instrument(skip(self))]
@@ -2214,27 +2016,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
return Err(Error::other("part result number err"));
}
if let Some(limit) = multipart_size_limit_from_metadata(&fi.metadata)? {
let mut total = 0_u64;
for part in &object_parts {
if part.error.is_some() {
return Err(Error::PartMissingOrCorrupt);
}
let part_size = u64::try_from(part.actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
total = total.checked_add(part_size).ok_or_else(|| {
Error::InvalidArgument(
"multipart upload".to_string(),
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
"logical size overflow".to_string(),
)
})?;
}
if total > limit {
return Err(Error::EntityTooLarge(total, limit));
}
rustfs_utils::http::metadata_compat::remove_str(&mut fi.metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE);
}
let mut checksum_type = rustfs_rio::ChecksumType::NONE;
if let Some(cs) = fi.metadata.get(rustfs_rio::RUSTFS_MULTIPART_CHECKSUM) {
@@ -3243,17 +3024,13 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
Ok(ObjectInfo::from_file_info(&fi, &commit_bucket, &commit_object, commit_is_versioned))
};
let result = if detach_commit_owner {
if detach_commit_owner {
tokio::spawn(commit)
.await
.map_err(|err| Error::other(format!("complete_multipart_upload commit task failed: {err}")))?
} else {
commit.await
};
if result.is_ok() {
remove_capped_multipart_staging_semaphore(&upload_id_path);
}
result
}
}
@@ -3319,34 +3096,6 @@ mod tests {
assert!(multipart_bucket_incarnation_id(&nil_metadata).is_err());
}
#[test]
fn multipart_size_limit_metadata_is_dual_key_and_fail_closed() {
let mut metadata = HashMap::new();
insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "100".to_string());
assert_eq!(multipart_size_limit_from_metadata(&metadata).unwrap(), Some(100));
metadata.insert("x-minio-internal-max-total-object-size".to_string(), "101".to_string());
assert!(multipart_size_limit_from_metadata(&metadata).is_err());
let mut invalid = HashMap::new();
insert_str(&mut invalid, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "-1".to_string());
assert!(multipart_size_limit_from_metadata(&invalid).is_err());
}
#[test]
fn multipart_size_admission_handles_boundaries_and_overflow() {
assert_eq!(admitted_multipart_size(90, 10, 100).unwrap(), 100);
assert!(matches!(
admitted_multipart_size(90, 11, 100),
Err(StorageError::EntityTooLarge(101, 100))
));
assert!(admitted_multipart_size(u64::MAX - 1, 1, u64::MAX).is_ok());
assert!(matches!(
admitted_multipart_size(u64::MAX, 1, u64::MAX),
Err(StorageError::InvalidArgument(_, _, _))
));
}
#[test]
fn multipart_bucket_incarnation_gate_accepts_only_current_or_same_lifetime_legacy_uploads() {
let expected = Uuid::new_v4();
+12 -11
View File
@@ -19,6 +19,7 @@
//! bounds are unchanged, and the impls reach shared primitives through the
//! SetDisks core (io_primitives) via inherent calls.
use ahash::AHashMap;
#[cfg(test)]
use super::super::MetadataCacheInvalidationProbe;
use super::super::{
@@ -1107,13 +1108,13 @@ fn is_restore_control_metadata(key: &str) -> bool {
.is_some_and(|remainder| remainder.is_empty())
}
fn restore_metadata_update_preserves_protected_metadata(
existing: &HashMap<String, String>,
replacement: &HashMap<String, String>,
fn restore_metadata_update_preserves_protected_metadata<S1: std::hash::BuildHasher, S2: std::hash::BuildHasher>(
existing: &HashMap<String, String, S1>,
replacement: &HashMap<String, String, S2>,
) -> bool {
let mut existing = existing.clone();
let mut existing: HashMap<String, String> = existing.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
clean_metadata(&mut existing);
let mut replacement = replacement.clone();
let mut replacement: HashMap<String, String> = replacement.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
clean_metadata(&mut replacement);
let existing_count = existing.keys().filter(|key| !is_restore_control_metadata(key)).count();
let replacement_count = replacement.keys().filter(|key| !is_restore_control_metadata(key)).count();
@@ -2211,9 +2212,9 @@ pub(in crate::set_disk) fn stored_replication_category_metadata(existing: &Objec
///
/// Returns whether `inbound` was modified. Callers must hold the object write
/// lock so the stored values compared here are the ones being replaced.
pub(in crate::set_disk) fn merge_replication_metadata_lww(
inbound: &mut HashMap<String, String>,
existing: &HashMap<String, String>,
pub(in crate::set_disk) fn merge_replication_metadata_lww<S1: std::hash::BuildHasher, S2: std::hash::BuildHasher>(
inbound: &mut HashMap<String, String, S1>,
existing: &HashMap<String, String, S2>,
opts: &ObjectOptions,
) -> bool {
use rustfs_utils::http::headers::{
@@ -2844,7 +2845,7 @@ impl SetDisks {
)));
}
fi.metadata = user_defined;
fi.metadata = user_defined.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
fi.mod_time = mod_time;
fi.size = w_size as i64;
fi.versioned = opts.versioned || opts.version_suspended;
@@ -6051,7 +6052,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
} else {
None
};
let mut replacement_metadata = (*src_info.user_defined).clone();
let mut replacement_metadata: AHashMap<String, String> = (*src_info.user_defined).iter().map(|(k, v)| (k.clone(), v.clone())).collect();
if let Some(part_checksums) = preserved_part_checksums {
rustfs_utils::http::insert_str(&mut replacement_metadata, rustfs_utils::http::SUFFIX_PART_CHECKSUMS, part_checksums);
}
@@ -7493,7 +7494,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str(),
X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str(),
] {
if let Some(value) = fi.metadata.lookup(header).filter(|value| !value.is_empty()) {
if let Some(value) = fi.metadata.get(header).filter(|value| !value.is_empty()) {
transition_meta.insert(header.to_ascii_lowercase(), value.to_string());
}
}
+16 -196
View File
@@ -99,8 +99,6 @@ fn preflight_startup_rpc_secret_with(
const LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES: usize = 6;
const LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY: Duration = Duration::from_secs(60 * 3);
const LOCAL_DECOMMISSION_RESUME_RETRY_DELAY: Duration = Duration::from_secs(30);
const REBALANCE_INITIAL_RESUME_DELAY: Duration = Duration::from_secs(10);
const REBALANCE_RESUME_RETRY_DELAY: Duration = Duration::from_secs(10);
fn should_retry_local_decommission_resume(err: &Error, attempt: usize) -> bool {
matches!(err, Error::ConfigNotFound) && attempt < LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES
@@ -110,12 +108,8 @@ fn should_retry_format_load(err: &Error) -> bool {
!matches!(err, Error::CorruptedFormat)
}
fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_resume_required: bool) -> bool {
rebalance_resume_required && !decommission_running
}
fn should_defer_rebalance_auto_start(distributed: bool, fleet_proof_available: bool) -> bool {
distributed && !fleet_proof_available
fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_meta_loaded: bool) -> bool {
rebalance_meta_loaded && !decommission_running
}
fn should_schedule_local_decommission_resume(
@@ -133,17 +127,6 @@ async fn wait_for_local_decommission_resume_delay(rx: &CancellationToken, delay:
}
}
async fn wait_for_rebalance_resume_delay(rx: &CancellationToken, delay: Duration) -> bool {
tokio::select! {
_ = rx.cancelled() => false,
_ = tokio::time::sleep(delay) => true,
}
}
async fn wait_for_rebalance_resume_retry(rx: &CancellationToken) -> bool {
wait_for_rebalance_resume_delay(rx, REBALANCE_RESUME_RETRY_DELAY).await
}
fn resolve_store_init_stage_result(result: Result<()>, stage: &str) -> Result<()> {
result.map_err(|err| Error::other(format!("store init failed during {stage}: {err}")))
}
@@ -300,71 +283,6 @@ async fn resume_local_decommission_after_init(store: Arc<ECStore>, rx: Cancellat
}
}
async fn resume_rebalance_after_init(store: Arc<ECStore>, rx: CancellationToken) {
if !wait_for_rebalance_resume_delay(&rx, REBALANCE_INITIAL_RESUME_DELAY).await {
return;
}
loop {
if rx.is_cancelled() {
return;
}
let resume_required = store
.rebalance_meta
.read()
.await
.as_ref()
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation);
if !resume_required {
return;
}
if should_defer_rebalance_auto_start(
store.ctx.is_dist_erasure().await,
crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_some(),
) {
if !wait_for_rebalance_resume_retry(&rx).await {
return;
}
continue;
}
match store.start_rebalance().await {
Ok(()) => return,
Err(err) if crate::core::pools::is_pool_activation_fleet_proof_error(&err) => {
warn!(
event = EVENT_ECSTORE_INIT_STATUS,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_STORE_INIT,
stage = "start_rebalance",
state = "retrying",
reason = "fleet_capability_proof_unavailable",
error = %err,
retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
"Retrying deferred rebalance auto-start"
);
if !wait_for_rebalance_resume_retry(&rx).await {
return;
}
}
Err(err) => {
error!(
event = EVENT_ECSTORE_INIT_STATUS,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_STORE_INIT,
stage = "start_rebalance",
state = "failed",
reason = "deferred_resume_failed",
error = %err,
"Failed to resume rebalance after store initialization"
);
return;
}
}
}
}
impl ECStore {
/// Validate topology and process storage-class overrides before any disk is opened.
pub fn validate_startup_storage_class(endpoint_pools: &EndpointServerPools) -> Result<()> {
@@ -656,49 +574,12 @@ impl ECStore {
}
resolve_store_init_stage_result(self.load_rebalance_meta().await, "load_rebalance_meta")?;
let rebalance_resume_required = {
let rebalance_meta = self.rebalance_meta.read().await;
rebalance_meta
.as_ref()
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
};
let rebalance_meta_loaded = self.rebalance_meta.read().await.is_some();
let decommission_running =
pool_meta_has_active_decommission(&installed_pool_meta) || self.is_decommission_running().await;
let distributed = self.ctx.is_dist_erasure().await;
let fleet_proof_available = crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_some();
let mut rebalance_auto_start_deferred = false;
if should_auto_start_rebalance_after_init(decommission_running, rebalance_resume_required) {
if should_defer_rebalance_auto_start(distributed, fleet_proof_available) {
rebalance_auto_start_deferred = true;
warn!(
event = EVENT_ECSTORE_INIT_STATUS,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_STORE_INIT,
stage = "start_rebalance",
state = "deferred",
reason = "fleet_capability_proof_unavailable",
retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
"Deferred rebalance auto-start until a live fleet capability proof is available"
);
} else if let Err(err) = self.start_rebalance().await {
if crate::core::pools::is_pool_activation_fleet_proof_error(&err) {
rebalance_auto_start_deferred = true;
warn!(
event = EVENT_ECSTORE_INIT_STATUS,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_STORE_INIT,
stage = "start_rebalance",
state = "deferred",
reason = "fleet_capability_proof_changed",
error = %err,
retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
"Deferred rebalance auto-start after the fleet capability proof changed"
);
} else {
return resolve_store_init_stage_result(Err(err), "start_rebalance");
}
}
} else if decommission_running && rebalance_resume_required {
if should_auto_start_rebalance_after_init(decommission_running, rebalance_meta_loaded) {
resolve_store_init_stage_result(self.start_rebalance().await, "start_rebalance")?;
} else if decommission_running && rebalance_meta_loaded {
warn!(
event = EVENT_ECSTORE_INIT_STATUS,
component = LOG_COMPONENT_ECSTORE,
@@ -735,13 +616,12 @@ impl ECStore {
.is_ok();
if should_schedule_local_decommission_resume(&local_pool_indices, pool_meta_replica_state, pool_meta_write_safe) {
let store = self.clone();
let decommission_rx = rx.clone();
tokio::spawn(async move {
if !wait_for_local_decommission_resume_delay(&decommission_rx, LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY).await {
if !wait_for_local_decommission_resume_delay(&rx, LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY).await {
return;
}
resume_local_decommission_after_init(store, decommission_rx, local_pool_indices).await;
resume_local_decommission_after_init(store, rx, local_pool_indices).await;
});
} else if !local_pool_indices.is_empty() {
error!(
@@ -768,11 +648,6 @@ impl ECStore {
info!("TierConfigMgr init error: {}", err);
}
if rebalance_auto_start_deferred {
let store = self.clone();
tokio::spawn(resume_rebalance_after_init(store, rx));
}
Ok(())
}
@@ -790,8 +665,7 @@ mod tests {
load_pool_meta_for_startup, persist_pool_meta_for_startup_if_safe, pool_first_endpoint_is_local,
pool_meta_has_active_decommission, preflight_startup_rpc_secret_with, resolve_startup_pool_defaults_with,
resolve_store_init_stage_result, save_validated_pool_meta_for_startup, should_auto_start_rebalance_after_init,
should_defer_rebalance_auto_start, should_retry_format_load, should_retry_local_decommission_resume,
wait_for_local_decommission_resume_delay,
should_retry_format_load, should_retry_local_decommission_resume, wait_for_local_decommission_resume_delay,
};
#[cfg(feature = "test-util")]
use crate::disk::DiskAPI;
@@ -1576,7 +1450,7 @@ mod tests {
}
#[test]
fn test_should_auto_start_rebalance_after_init_allows_active_rebalance_without_decommission() {
fn test_should_auto_start_rebalance_after_init_allows_loaded_rebalance_without_decommission() {
assert!(should_auto_start_rebalance_after_init(false, true));
}
@@ -1586,17 +1460,10 @@ mod tests {
}
#[test]
fn test_should_auto_start_rebalance_after_init_rejects_terminal_or_missing_rebalance() {
fn test_should_auto_start_rebalance_after_init_rejects_missing_rebalance_meta() {
assert!(!should_auto_start_rebalance_after_init(false, false));
}
#[test]
fn test_should_defer_rebalance_auto_start_only_without_distributed_fleet_proof() {
assert!(should_defer_rebalance_auto_start(true, false));
assert!(!should_defer_rebalance_auto_start(true, true));
assert!(!should_defer_rebalance_auto_start(false, false));
}
#[test]
fn test_store_init_recovery_skips_rebalance_when_decommission_metadata_is_active() {
let pool_meta = init_test_pool_meta(Some(PoolDecommissionInfo {
@@ -1606,69 +1473,22 @@ mod tests {
canceled: false,
..Default::default()
}));
let rebalance_meta = Some(RebalanceMeta {
pool_stats: vec![RebalanceStats {
participating: true,
info: RebalanceInfo {
status: RebalStatus::Started,
..Default::default()
},
..Default::default()
}],
..Default::default()
});
let rebalance_meta = Some(RebalanceMeta::default());
assert!(!should_auto_start_rebalance_after_init(
pool_meta_has_active_decommission(&pool_meta),
rebalance_meta
.as_ref()
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
rebalance_meta.is_some()
));
}
#[test]
fn test_store_init_recovery_allows_active_rebalance_without_decommission() {
fn test_store_init_recovery_allows_rebalance_when_only_rebalance_metadata_exists() {
let pool_meta = init_test_pool_meta(None);
let rebalance_meta = Some(RebalanceMeta {
pool_stats: vec![RebalanceStats {
participating: true,
info: RebalanceInfo {
status: RebalStatus::Started,
..Default::default()
},
..Default::default()
}],
..Default::default()
});
let rebalance_meta = Some(RebalanceMeta::default());
assert!(should_auto_start_rebalance_after_init(
pool_meta_has_active_decommission(&pool_meta),
rebalance_meta
.as_ref()
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
));
}
#[test]
fn test_store_init_recovery_skips_completed_rebalance_metadata() {
let pool_meta = init_test_pool_meta(None);
let rebalance_meta = Some(RebalanceMeta {
pool_stats: vec![RebalanceStats {
participating: true,
info: RebalanceInfo {
status: RebalStatus::Completed,
..Default::default()
},
..Default::default()
}],
..Default::default()
});
assert!(!should_auto_start_rebalance_after_init(
pool_meta_has_active_decommission(&pool_meta),
rebalance_meta
.as_ref()
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
rebalance_meta.is_some()
));
}
+3 -3
View File
@@ -26,18 +26,18 @@ static STRICT_BUCKET_NAME_REGEX: LazyLock<Regex> =
static NON_STRICT_BUCKET_NAME_REGEX: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^[A-Za-z0-9][A-Za-z0-9\.\-_:]{1,61}[A-Za-z0-9]$").expect("valid non-strict bucket name regex"));
pub fn clean_metadata(metadata: &mut HashMap<String, String>) {
pub fn clean_metadata<S: std::hash::BuildHasher>(metadata: &mut HashMap<String, String, S>) {
remove_standard_storage_class(metadata);
clean_metadata_keys(metadata, &["md5Sum", "etag", "expires", AMZ_OBJECT_TAGGING, "last-modified"]);
}
pub fn remove_standard_storage_class(metadata: &mut HashMap<String, String>) {
pub fn remove_standard_storage_class<S: std::hash::BuildHasher>(metadata: &mut HashMap<String, String, S>) {
if metadata.get(AMZ_STORAGE_CLASS) == Some(&STANDARD.to_string()) {
metadata.remove(AMZ_STORAGE_CLASS);
}
}
pub fn clean_metadata_keys(metadata: &mut HashMap<String, String>, key_names: &[&str]) {
pub fn clean_metadata_keys<S: std::hash::BuildHasher>(metadata: &mut HashMap<String, String, S>, key_names: &[&str]) {
for key in key_names {
metadata.remove(key.to_owned());
}
+3
View File
@@ -51,6 +51,9 @@ s3s = { workspace = true, features = ["minio"] }
regex.workspace = true
arc-swap.workspace = true
# High-performance hashing
ahash = { workspace = true, features = ["serde"] }
[dev-dependencies]
criterion = { workspace = true, features = ["html_reports"] }
tempfile = { workspace = true }
+9 -8
View File
@@ -22,6 +22,7 @@ use rustfs_utils::http::{
contains_key_str, get_consistent_str, get_str, has_internal_suffix, insert_str, is_encryption_metadata_key,
starts_with_ignore_ascii_case,
};
use ahash::AHashMap;
use s3s::dto::{RestoreStatus, Timestamp};
use s3s::header::X_AMZ_RESTORE;
use serde::de::{self, MapAccess, SeqAccess, Visitor, value::MapAccessDeserializer};
@@ -67,7 +68,7 @@ pub struct ObjectPartInfo {
// Index holds the index of the part in the erasure coding
pub index: Option<Bytes>,
// Checksums holds checksums of the part
pub checksums: Option<HashMap<String, String>>,
pub checksums: Option<AHashMap<String, String>>,
pub error: Option<String>,
}
@@ -268,7 +269,7 @@ pub struct FileInfo {
pub mode: Option<u32>,
// WrittenByVersion is the unix time stamp of the version that created this version of the object
pub written_by_version: Option<u64>,
pub metadata: HashMap<String, String>,
pub metadata: AHashMap<String, String>,
pub parts: Vec<ObjectPartInfo>,
pub erasure: ErasureInfo,
// MarkDeleted marks this version as deleted
@@ -301,7 +302,7 @@ fn is_sensitive_metadata_key(key: &str) -> bool {
.any(|prefix| starts_with_ignore_ascii_case(key, prefix))
}
struct RedactedMetadata<'a>(&'a HashMap<String, String>);
struct RedactedMetadata<'a>(&'a AHashMap<String, String>);
impl std::fmt::Debug for RedactedMetadata<'_> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
@@ -425,7 +426,7 @@ struct FileInfoMapDef {
size: i64,
mode: Option<u32>,
written_by_version: Option<u64>,
metadata: HashMap<String, String>,
metadata: AHashMap<String, String>,
parts: Vec<ObjectPartInfo>,
erasure: ErasureInfo,
mark_deleted: bool,
@@ -1079,7 +1080,7 @@ impl FileInfo {
mod_time: Option<OffsetDateTime>,
actual_size: i64,
index: Option<Bytes>,
checksums: Option<HashMap<String, String>>,
checksums: Option<AHashMap<String, String>>,
) {
let part = ObjectPartInfo {
etag,
@@ -1457,7 +1458,7 @@ pub fn parse_restore_obj_status(restore_hdr: &str) -> Result<RestoreStatus> {
Err(Error::other(ERR_RESTORE_HDR_MALFORMED))
}
pub fn is_restored_object_on_disk(meta: &HashMap<String, String>) -> bool {
pub fn is_restored_object_on_disk<S: std::hash::BuildHasher>(meta: &HashMap<String, String, S>) -> bool {
if let Some(restore_hdr) = meta.get(X_AMZ_RESTORE.as_str())
&& let Ok(restore_status) = parse_restore_obj_status(restore_hdr)
{
@@ -2135,7 +2136,7 @@ mod tests {
-1_000_000i64..=1_000_000i64,
optional_timestamp_strategy(),
proptest::option::of(bytes_strategy(16)),
proptest::option::of(hash_map(small_string_strategy(), small_string_strategy(), 0..=3)),
proptest::option::of(hash_map(small_string_strategy(), small_string_strategy(), 0..=3).prop_map(|m| m.into_iter().collect::<AHashMap<String, String>>())),
proptest::option::of(small_string_strategy()),
)
.prop_map(|(etag, number, size, actual_size, mod_time, index, checksums, error)| ObjectPartInfo {
@@ -2170,7 +2171,7 @@ mod tests {
-1_000_000i64..=1_000_000i64,
proptest::option::of(any::<u32>()),
proptest::option::of(any::<u64>()),
hash_map(small_string_strategy(), small_string_strategy(), 0..=4),
hash_map(small_string_strategy(), small_string_strategy(), 0..=4).prop_map(|m| m.into_iter().collect::<AHashMap<String, String>>()),
vec(object_part_info_strategy(), 0..=3),
erasure_info_strategy(),
any::<bool>(),
+2 -2
View File
@@ -174,10 +174,10 @@ fn valid_target_delete_marker_version(arn: &str, version_id: &str) -> bool {
/// included in the quorum hash, so such a divergence does surface — but as a
/// quorum failure on an otherwise healthy object, which is not a state worth
/// reaching. Merge the RPC metadata carrier instead, and only ever insert.
fn persist_target_delete_marker_versions(
fn persist_target_delete_marker_versions<S: std::hash::BuildHasher>(
meta_sys: &mut HashMap<String, Vec<u8>>,
versions: &HashMap<String, String>,
transport_metadata: &HashMap<String, String>,
transport_metadata: &HashMap<String, String, S>,
) {
let mut bounded = BTreeMap::new();
// A corrupt carrier means the dual internal prefixes disagreed. Do not merge
+1 -1
View File
@@ -181,7 +181,7 @@ mod tests {
data_dir: Some(data_dir),
size: 64 * 1024,
mod_time: Some(OffsetDateTime::now_utc()),
metadata,
metadata: metadata.into_iter().collect(),
erasure: ErasureInfo {
algorithm: ErasureAlgo::ReedSolomon.to_string(),
data_blocks: 4,
+11 -11
View File
@@ -26,7 +26,7 @@ use super::msgp_decode::{
PrependByteReader, prealloc_hint, read_exact_vec, read_nil_or_array_len, read_nil_or_map_len, skip_msgp_value,
};
use super::*;
use crate::{ChecksumInfo, TransitionVersionState};
use crate::{AHashMap, ChecksumInfo, TransitionVersionState};
use rustfs_utils::HashAlgorithm;
use rustfs_utils::http::{
RUSTFS_INTERNAL_PREFIX, SUFFIX_CRC, SUFFIX_FREE_VERSION, SUFFIX_INLINE_DATA, SUFFIX_PART_CHECKSUMS, SUFFIX_PURGESTATUS,
@@ -377,7 +377,7 @@ impl<'a> DerivedInternalMetadata<'a> {
}
}
struct UniquePartChecksums(HashMap<String, String>);
struct UniquePartChecksums(AHashMap<String, String>);
impl<'de> serde::Deserialize<'de> for UniquePartChecksums {
fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
@@ -397,7 +397,7 @@ impl<'de> serde::Deserialize<'de> for UniquePartChecksums {
where
A: serde::de::SeqAccess<'de>,
{
let mut checksums = HashMap::with_capacity(seq.size_hint().unwrap_or_default());
let mut checksums = AHashMap::with_capacity(seq.size_hint().unwrap_or_default());
while let Some((key, value)) = seq.next_element::<(String, String)>()? {
if checksums.insert(key, value).is_some() {
return Err(serde::de::Error::custom("duplicate part checksum name"));
@@ -1477,7 +1477,7 @@ pub struct MetaObjectV1 {
#[serde(rename = "Erasure")]
pub erasure: MetaObjectV1Erasure,
#[serde(rename = "Meta")]
pub meta: HashMap<String, String>,
pub meta: AHashMap<String, String>,
#[serde(rename = "Parts")]
pub parts: Vec<MetaObjectV1Part>,
#[serde(rename = "VersionID")]
@@ -1543,7 +1543,7 @@ pub struct MetaObjectV1Part {
#[serde(rename = "i")]
pub index: Option<Bytes>,
#[serde(rename = "crc")]
pub checksums: Option<HashMap<String, String>>,
pub checksums: Option<AHashMap<String, String>>,
#[serde(rename = "err")]
pub error: Option<String>,
}
@@ -1887,7 +1887,7 @@ impl MetaObjectV1Part {
"i" => self.index = Some(Bytes::from(read_msgp_bin(rd)?)),
"crc" => {
let len = rmp::decode::read_map_len(rd)? as usize;
let mut checksums = HashMap::with_capacity(prealloc_hint(len));
let mut checksums = AHashMap::with_capacity(prealloc_hint(len));
for _ in 0..len {
checksums.insert(read_msgp_string(rd)?, read_msgp_string(rd)?);
}
@@ -2570,7 +2570,7 @@ impl MetaObject {
Vec::new()
};
let mut metadata = HashMap::with_capacity(self.meta_user.len() + self.meta_sys.len());
let mut metadata = AHashMap::with_capacity(self.meta_user.len() + self.meta_sys.len());
for (k, v) in &self.meta_user {
if k == AMZ_META_UNENCRYPTED_CONTENT_LENGTH || k == AMZ_META_UNENCRYPTED_CONTENT_MD5 {
continue;
@@ -2861,7 +2861,7 @@ impl From<FileInfo> for MetaObject {
}
}
fn get_internal_replication_state(metadata: &HashMap<String, String>) -> Option<ReplicationState> {
fn get_internal_replication_state<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> Option<ReplicationState> {
let mut rs = ReplicationState::default();
let mut has = false;
@@ -2942,7 +2942,7 @@ impl MetaDeleteMarker {
}
pub fn into_fileinfo(&self, volume: &str, path: &str, _all_parts: bool) -> Result<FileInfo> {
let metadata = self
let metadata: AHashMap<String, String> = self
.meta_sys
.clone()
.into_iter()
@@ -5500,10 +5500,10 @@ mod tests {
/// entirely, silently dropping the whole legacy body on re-marshal.
#[test]
fn legacy_version_body_round_trips_through_encode() {
let mut meta = HashMap::new();
let mut meta = AHashMap::new();
meta.insert("content-type".to_string(), "application/octet-stream".to_string());
let mut crc = HashMap::new();
let mut crc = AHashMap::new();
crc.insert("crc32c".to_string(), "deadbeef".to_string());
let legacy = MetaObjectV1 {
+6
View File
@@ -22,6 +22,12 @@ mod replication;
pub mod test_data;
/// High-performance HashMap type alias using ahash instead of SipHash.
pub type AHashMap<K, V> = ahash::AHashMap<K, V>;
/// High-performance HashSet type alias using ahash.
pub type AHashSet<K> = ahash::AHashSet<K>;
pub use error::*;
pub use fileinfo::*;
pub use filemeta::*;
+6 -6
View File
@@ -14,8 +14,8 @@
use crate::filemeta::msgp_decode::MAX_MSGP_ELEMENT_SIZE;
use crate::{
Error, FileInfo, FileInfoOpts, FileInfoVersions, FileMeta, FileMetaShallowVersion, Result, VersionType, get_file_info,
merge_file_meta_versions, merge_file_meta_versions_with_write_quorum,
AHashMap, Error, FileInfo, FileInfoOpts, FileInfoVersions, FileMeta, FileMetaShallowVersion, Result, VersionType,
get_file_info, merge_file_meta_versions, merge_file_meta_versions_with_write_quorum,
};
use arc_swap::ArcSwapOption;
use rmp::Marker;
@@ -1676,7 +1676,7 @@ mod tests {
}
fn metacache_entry_with_mod_time(mod_time: OffsetDateTime, etag: &str) -> MetaCacheEntry {
let mut metadata = HashMap::new();
let mut metadata = AHashMap::new();
metadata.insert("etag".to_string(), etag.to_string());
let mut meta = FileMeta::new();
@@ -1705,7 +1705,7 @@ mod tests {
data_blocks: usize,
parity_blocks: usize,
) -> MetaCacheEntry {
let mut metadata = HashMap::new();
let mut metadata = AHashMap::new();
metadata.insert("etag".to_string(), etag.to_string());
let mut fi = FileInfo::new("object", data_blocks, parity_blocks);
@@ -1730,7 +1730,7 @@ mod tests {
fn metacache_entry_with_erasure_versions(versions: &[(OffsetDateTime, &str, usize, usize)]) -> MetaCacheEntry {
let mut meta = FileMeta::new();
for (idx, (mod_time, etag, data_blocks, parity_blocks)) in versions.iter().enumerate() {
let mut metadata = HashMap::new();
let mut metadata = AHashMap::new();
metadata.insert("etag".to_string(), (*etag).to_string());
let mut fi = FileInfo::new("object", *data_blocks, *parity_blocks);
@@ -1776,7 +1776,7 @@ mod tests {
/// Build an entry holding a single object version with an explicit version id
/// and mod_time, so a set of these can model DISJOINT per-disk version sets.
fn metacache_entry_single_version(version_u128: u128, mod_time: OffsetDateTime, etag: &str) -> MetaCacheEntry {
let mut metadata = HashMap::new();
let mut metadata = AHashMap::new();
metadata.insert("etag".to_string(), etag.to_string());
let mut fi = FileInfo::new("object", 4, 2);
-4
View File
@@ -81,7 +81,6 @@ pub enum StorageErrorCode {
InsufficientWriteQuorum,
PreconditionFailed,
EntityTooSmall,
EntityTooLarge,
InvalidRangeSpec,
NotModified,
InvalidPartNumber,
@@ -170,7 +169,6 @@ impl StorageErrorCode {
Self::InsufficientWriteQuorum => 0x3A,
Self::PreconditionFailed => 0x3B,
Self::EntityTooSmall => 0x3C,
Self::EntityTooLarge => 0x56,
Self::InvalidRangeSpec => 0x3D,
Self::NotModified => 0x3E,
Self::InvalidPartNumber => 0x3F,
@@ -259,7 +257,6 @@ impl StorageErrorCode {
0x3A => Some(Self::InsufficientWriteQuorum),
0x3B => Some(Self::PreconditionFailed),
0x3C => Some(Self::EntityTooSmall),
0x56 => Some(Self::EntityTooLarge),
0x3D => Some(Self::InvalidRangeSpec),
0x3E => Some(Self::NotModified),
0x3F => Some(Self::InvalidPartNumber),
@@ -353,7 +350,6 @@ mod tests {
(StorageErrorCode::InsufficientWriteQuorum, 0x3A),
(StorageErrorCode::PreconditionFailed, 0x3B),
(StorageErrorCode::EntityTooSmall, 0x3C),
(StorageErrorCode::EntityTooLarge, 0x56),
(StorageErrorCode::InvalidRangeSpec, 0x3D),
(StorageErrorCode::NotModified, 0x3E),
(StorageErrorCode::InvalidPartNumber, 0x3F),
+3
View File
@@ -58,6 +58,9 @@ transform-stream = { workspace = true, optional = true }
url = { workspace = true, optional = true }
zstd = { workspace = true, optional = true }
# High-performance hashing
ahash = { workspace = true, optional = true }
[dev-dependencies]
criterion = { workspace = true, features = ["html_reports"] }
tempfile = { workspace = true }
+2 -2
View File
@@ -89,7 +89,7 @@ pub fn is_object_encryption_marker(key: &str) -> bool {
}
/// Reads the logical object size recorded by encryption metadata.
pub fn get_object_encryption_original_size(metadata: &std::collections::HashMap<String, String>) -> std::io::Result<Option<i64>> {
pub fn get_object_encryption_original_size<S: std::hash::BuildHasher>(metadata: &std::collections::HashMap<String, String, S>) -> std::io::Result<Option<i64>> {
let actual_size = super::get_str(metadata, super::SUFFIX_ACTUAL_SIZE);
let size = get_case_insensitive(metadata, RUSTFS_ENCRYPTION_ORIGINAL_SIZE)
.or_else(|| get_case_insensitive(metadata, SSEC_ORIGINAL_SIZE))
@@ -103,7 +103,7 @@ pub fn get_object_encryption_original_size(metadata: &std::collections::HashMap<
.map_err(|error| std::io::Error::other(format!("Failed to parse encryption original size: {error}")))
}
fn get_case_insensitive<'a>(metadata: &'a std::collections::HashMap<String, String>, key: &str) -> Option<&'a str> {
fn get_case_insensitive<'a, S: std::hash::BuildHasher>(metadata: &'a std::collections::HashMap<String, String, S>, key: &str) -> Option<&'a str> {
metadata.get(key).map(String::as_str).or_else(|| {
metadata
.iter()
+7 -9
View File
@@ -44,8 +44,6 @@ pub const SUFFIX_COMPRESSION: &str = "compression";
pub const SUFFIX_REPLICATION_PRESERVE_CIPHERTEXT: &str = "replication-preserve-ciphertext";
pub const SUFFIX_COMPRESSION_SIZE: &str = "compression-size";
pub const SUFFIX_ACTUAL_SIZE: &str = "actual-size";
/// Maximum logical object size for a capability-bound multipart upload.
pub const SUFFIX_MAX_TOTAL_OBJECT_SIZE: &str = "max-total-object-size";
pub const SUFFIX_ACTUAL_OBJECT_SIZE: &str = "actual-object-size";
/// Used by replication; key stored with capital A
pub const SUFFIX_ACTUAL_OBJECT_SIZE_CAP: &str = "Actual-Object-Size";
@@ -184,13 +182,13 @@ pub fn internal_key_rustfs(suffix: &str) -> String {
// === String type (FileInfo.metadata, user_defined) ===
pub fn insert_str(map: &mut HashMap<String, String>, suffix: &str, value: String) {
pub fn insert_str<S: std::hash::BuildHasher>(map: &mut HashMap<String, String, S>, suffix: &str, value: String) {
let (k1, k2) = both_keys(suffix);
map.insert(k1, value.clone());
map.insert(k2, value);
}
pub fn get_str(map: &HashMap<String, String>, suffix: &str) -> Option<String> {
pub fn get_str<S: std::hash::BuildHasher>(map: &HashMap<String, String, S>, suffix: &str) -> Option<String> {
if let Some(v) = with_internal_key(RUSTFS_INTERNAL_PREFIX, suffix, |k1| map.get(k1).cloned()) {
return Some(v);
}
@@ -204,7 +202,7 @@ pub fn get_str(map: &HashMap<String, String>, suffix: &str) -> Option<String> {
.map(|(_, value)| value.clone())
}
fn get_consistent_value<'a, V: AsRef<[u8]>>(map: &'a HashMap<String, V>, suffix: &str) -> Option<&'a V> {
fn get_consistent_value<'a, V: AsRef<[u8]>, S: std::hash::BuildHasher>(map: &'a HashMap<String, V, S>, suffix: &str) -> Option<&'a V> {
let (rustfs_key, minio_key) = both_keys(suffix);
let mut value = None;
for (key, candidate) in map {
@@ -222,11 +220,11 @@ fn get_consistent_value<'a, V: AsRef<[u8]>>(map: &'a HashMap<String, V>, suffix:
/// Returns a non-empty value when every compatibility key present for `suffix` agrees.
/// A single RustFS or MinIO key is accepted for backward compatibility; conflicting or empty
/// values return `None` so callers at destructive boundaries can fail closed.
pub fn get_consistent_str<'a>(map: &'a HashMap<String, String>, suffix: &str) -> Option<&'a str> {
pub fn get_consistent_str<'a, S: std::hash::BuildHasher>(map: &'a HashMap<String, String, S>, suffix: &str) -> Option<&'a str> {
get_consistent_value(map, suffix).map(String::as_str)
}
pub fn contains_key_str(map: &HashMap<String, String>, suffix: &str) -> bool {
pub fn contains_key_str<S: std::hash::BuildHasher>(map: &HashMap<String, String, S>, suffix: &str) -> bool {
if with_internal_key(RUSTFS_INTERNAL_PREFIX, suffix, |k1| map.contains_key(k1)) {
return true;
}
@@ -238,7 +236,7 @@ pub fn contains_key_str(map: &HashMap<String, String>, suffix: &str) -> bool {
.any(|key| key.eq_ignore_ascii_case(&k1) || key.eq_ignore_ascii_case(&k2))
}
pub fn remove_str(map: &mut HashMap<String, String>, suffix: &str) {
pub fn remove_str<S: std::hash::BuildHasher>(map: &mut HashMap<String, String, S>, suffix: &str) {
with_internal_key(RUSTFS_INTERNAL_PREFIX, suffix, |k1| map.remove(k1));
with_internal_key(MINIO_INTERNAL_PREFIX, suffix, |k2| map.remove(k2));
let (k1, k2) = both_keys(suffix);
@@ -287,7 +285,7 @@ pub fn strip_internal_prefix_preserving_case(key: &str) -> Option<&str> {
/// Reads the bounded per-target delete-marker version map in one metadata scan.
/// The boolean is set when matching metadata is malformed or compatibility keys disagree.
pub fn target_delete_marker_versions(map: &HashMap<String, String>) -> (HashMap<String, String>, bool) {
pub fn target_delete_marker_versions<S: std::hash::BuildHasher>(map: &HashMap<String, String, S>) -> (HashMap<String, String>, bool) {
const MAX_ENTRIES: usize = 1_000;
const MAX_ARN_LEN: usize = 1_024;
const MAX_VERSION_ID_LEN: usize = 1_024;
+3 -3
View File
@@ -81,7 +81,7 @@ Scope and exemptions:
- **SSE-KMS only.** SSE-S3 wraps its data key with a server-owned key the caller never names, and SSE-C never reaches KMS; both are exempt, matching AWS.
- **The resolved key**, not the header. A bucket default encryption rule naming a KMS key is authorized the same way an explicit `x-amz-server-side-encryption-aws-kms-key-id` header is.
- **Anonymous requests are denied.** An anonymous caller has no identity policy and therefore holds no `kms` grants, so under enforcement every anonymous read or write of an SSE-KMS object fails with `AccessDenied` — even when a bucket policy makes the bucket public. This matches AWS, where anonymous requests cannot use SSE-KMS objects at all, and it keeps the per-key gate meaningful: were anonymous requests exempt, any denied identity could bypass the gate on a public bucket by simply dropping its credentials. **A public bucket serving SSE-KMS objects is incompatible with enforcement** — serve public content unencrypted or under SSE-S3 instead. With enforcement off (the default), anonymous access to SSE-KMS objects remains governed by bucket policy alone. The server warns once per process when it first denies an anonymous request; per-request denials appear on audit entries (`kmsOutcome=failure`, `kmsErrorClass=access_denied`, empty requester identity) and at debug level.
- **Anonymous requests are exempt.** They have no identity policy to evaluate, and denying them would break public buckets holding SSE-KMS objects. They remain governed by bucket policy.
- **Internal work is exempt.** Replication, lifecycle transitions, healing and the scanner run as the system principal.
- **Authorization runs before key state is checked**, so a denial cannot be used to probe whether a key exists, is disabled, or is pending deletion. The response is always `AccessDenied`.
- **Multipart uploads are authorized at create time**, where the session data key is generated. Part uploads and completion reuse that envelope and are not re-authorized against the destination key.
@@ -89,13 +89,13 @@ Scope and exemptions:
## Migration
Data-path enforcement is **off by default** because it changes the outcome of requests that succeed today: an identity holding only `s3:PutObject` can currently encrypt under any key. Turning it on without preparing policies will produce `AccessDenied` on working workloads.
Data-path enforcement is **off by default in this release** because it changes the outcome of requests that succeed today: an identity holding only `s3:PutObject` can currently encrypt under any key. Turning it on without preparing policies will produce `AccessDenied` on working workloads.
```bash
RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY=true
```
The server logs the configured mode once at startup, and warns while enforcement is off. Enforcement stays opt-in: there is no roadmap to flip the default.
The server logs the configured mode once at startup, and warns while enforcement is off. A later release defaults it to enabled.
Recommended sequence:
@@ -1,55 +0,0 @@
# Presigned multipart total-size limit
RustFS V2 supports an optional capability on a signed or SigV4-presigned
`CreateMultipartUpload` request:
```text
x-rustfs-max-total-object-size=<unsigned 64-bit integer>
```
The backend must include the parameter before calculating the SigV4
signature. It is part of the canonical query and cannot be added, removed, or
changed by the browser. RustFS stores the verified limit in the multipart
upload session and applies it to every `UploadPart` and to
`CompleteMultipartUpload`.
Backend pseudocode (the custom query must be present before signing):
```text
uri = "/photos/archive.zip?uploads"
uri += "&x-rustfs-max-total-object-size=104857600"
presigned_url = sigv4_presign("POST", uri, credentials)
# Return presigned_url to the browser. Never append the parameter afterwards.
```
The resulting flow is:
1. The backend signs `CreateMultipartUpload?...&x-rustfs-max-total-object-size=104857600`.
2. RustFS verifies the SigV4 request and persists the limit with the upload ID.
3. The browser uploads parts using the returned upload ID.
4. RustFS rejects a part whose declared logical size would exceed the remaining
budget and rejects completion if the server-side part metadata exceeds the
limit.
The limit is measured in logical object bytes (`actual_size`), not erasure,
encryption, or compression bytes. Replacing an existing part uses replacement
semantics: the old part size is removed before the new part size is admitted.
Unknown-length parts are rejected for capped sessions rather than buffered
without a bound. Capped parts are admitted under an upload-wide write lock
before temporary shards are created and use a per-upload staging permit to
bound local in-flight data. The distributed lock is released while the body is
read and reacquired for the final check/rename, so Complete and Abort are not
blocked behind a slow upload. The normal request-body stall timeout releases
the staging permit when a client stops sending.
The parameter is accepted only on `CreateMultipartUpload`. Supplying it on
`UploadPart`, `CompleteMultipartUpload`, `AbortMultipartUpload`, listing, or
copy operations returns `InvalidRequest`; those requests use the persisted
session state. A multipart upload created without this parameter remains
unlimited for backward compatibility. The V1 single-request capability
(`x-rustfs-max-content-length`) is independent and is not a multipart limit.
Because enforcement happens in the multipart data plane, every node that may
receive requests for a capped upload must run the V2 implementation. During a
rolling upgrade, route capped uploads only to upgraded nodes; older nodes treat
the internal metadata as unknown and cannot enforce the limit.
-5
View File
@@ -119,11 +119,6 @@ pull-request gate.
Use an exact preview tag for end-to-end release rehearsal. Manual dispatches
are backfill/debug paths and do not prove the automatic `workflow_run` chain.
A preview Release is internal validation state, not a deliverable: after the
final tag's release is published, `cleanup-preview-releases` deletes every
`<target>-preview.<N>` Release for that target. The tags themselves are kept, so
the validated commit stays traceable.
## Evidence requirements
A green check is useful only when it proves the intended behavior ran:
+18 -120
View File
@@ -67,7 +67,6 @@ use super::storage_api::multipart_usecase::sse::{
use super::storage_api::multipart_usecase::{
StorageObjectInfo as ObjectInfo, StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader,
};
use crate::app::object::{guard_put_object_body_read_timeout, put_object_body_read_timeout};
use crate::app::object_data_cache::{
ObjectDataCacheAdapter, invalidate_object_data_cache_after_complete_multipart_success,
invalidate_object_data_cache_before_mutation,
@@ -79,11 +78,7 @@ use crate::app::object_usecase::{
use crate::app::runtime_sources::{
AppContext, current_app_context, current_object_data_cache_for_context, current_object_store_handle_for_context,
};
use crate::auth::{
VerifiedPresignedRequest, VerifiedSigV4Request, parse_presigned_multipart_max_total_object_size,
reject_presigned_multipart_max_total_object_size_for_other_operation,
reject_presigned_put_max_content_length_for_other_operation,
};
use crate::auth::{VerifiedPresignedRequest, reject_presigned_put_max_content_length_for_other_operation};
use crate::capacity::record_capacity_write;
use crate::error::ApiError;
use crate::table_catalog;
@@ -97,9 +92,8 @@ use rustfs_utils::CompressionAlgorithm;
#[cfg(test)]
use rustfs_utils::http::insert_header;
use rustfs_utils::http::{
SUFFIX_MAX_TOTAL_OBJECT_SIZE, SUFFIX_REPLICATION_PRESERVE_CIPHERTEXT, SUFFIX_REPLICATION_STATUS,
SUFFIX_REPLICATION_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_REQUEST, contains_key_str, get_consistent_str, get_header,
get_source_scheme,
SUFFIX_REPLICATION_PRESERVE_CIPHERTEXT, SUFFIX_REPLICATION_STATUS, SUFFIX_REPLICATION_TIMESTAMP,
SUFFIX_SOURCE_REPLICATION_REQUEST, contains_key_str, get_header, get_source_scheme,
headers::{AMZ_DECODED_CONTENT_LENGTH, AMZ_OBJECT_TAGGING, AMZ_STORAGE_CLASS},
insert_str,
};
@@ -114,7 +108,6 @@ use s3s::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
use std::collections::{HashMap, HashSet};
use std::str::FromStr;
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::RwLock;
use tokio_util::io::StreamReader;
use tracing::{instrument, warn};
@@ -233,22 +226,6 @@ fn create_multipart_upload_metadata(
metadata
}
fn multipart_max_total_object_size(metadata: &HashMap<String, String>) -> S3Result<Option<u64>> {
if !contains_key_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE) {
return Ok(None);
}
let value = get_consistent_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE).ok_or_else(|| {
S3Error::with_message(
S3ErrorCode::InvalidRequest,
"multipart size capability metadata is missing or inconsistent".to_string(),
)
})?;
value.parse::<u64>().map(Some).map_err(|_| {
S3Error::with_message(S3ErrorCode::InvalidRequest, "multipart size capability metadata is invalid".to_string())
})
}
/// A multipart session advertises disk compression only when the staged-rollout
/// switch (`RUSTFS_COMPRESSION_MULTIPART_ENABLED`) is on, the object key/headers
/// qualify, AND the session is not an SSE-C ciphertext-passthrough replication
@@ -421,11 +398,6 @@ impl DefaultMultipartUsecase {
&self,
req: S3Request<AbortMultipartUploadInput>,
) -> S3Result<S3Response<AbortMultipartUploadOutput>> {
reject_presigned_multipart_max_total_object_size_for_other_operation(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedSigV4Request>().is_some(),
)?;
reject_presigned_put_max_content_length_for_other_operation(
&req.headers,
req.uri.query(),
@@ -472,11 +444,6 @@ impl DefaultMultipartUsecase {
&self,
req: S3Request<CompleteMultipartUploadInput>,
) -> S3Result<S3Response<CompleteMultipartUploadOutput>> {
reject_presigned_multipart_max_total_object_size_for_other_operation(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedSigV4Request>().is_some(),
)?;
reject_presigned_put_max_content_length_for_other_operation(
&req.headers,
req.uri.query(),
@@ -785,11 +752,6 @@ impl DefaultMultipartUsecase {
&self,
req: S3Request<CreateMultipartUploadInput>,
) -> S3Result<S3Response<CreateMultipartUploadOutput>> {
let multipart_max_total_object_size = parse_presigned_multipart_max_total_object_size(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedSigV4Request>().is_some(),
)?;
reject_presigned_put_max_content_length_for_other_operation(
&req.headers,
req.uri.query(),
@@ -845,9 +807,6 @@ impl DefaultMultipartUsecase {
)?;
let mut metadata = create_multipart_upload_metadata(input_metadata, &req.headers, tagging, storage_class.as_ref());
if let Some(limit) = multipart_max_total_object_size {
insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, limit.to_string());
}
let has_explicit_object_lock_retention = object_lock_mode.is_some()
|| object_lock_retain_until_date.is_some()
@@ -1019,11 +978,6 @@ impl DefaultMultipartUsecase {
#[instrument(level = "debug", skip(self, req))]
#[hotpath::measure(impl_type = "MultipartUsecase")]
pub async fn execute_upload_part(&self, req: S3Request<UploadPartInput>) -> S3Result<S3Response<UploadPartOutput>> {
reject_presigned_multipart_max_total_object_size_for_other_operation(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedSigV4Request>().is_some(),
)?;
reject_presigned_put_max_content_length_for_other_operation(
&req.headers,
req.uri.query(),
@@ -1052,40 +1006,6 @@ impl DefaultMultipartUsecase {
let mut size = resolve_upload_part_size(&req.headers, content_length)?;
let mut body_stream = body.ok_or_else(|| s3_error!(IncompleteBody))?;
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let fi = store
.get_multipart_info(&bucket, &key, &upload_id, &opts)
.await
.map_err(ApiError::from)?;
let max_total_object_size = multipart_max_total_object_size(&fi.user_defined)?;
if max_total_object_size.is_some() && size.is_some_and(|size| size < 0) {
return Err(S3Error::new(S3ErrorCode::UnexpectedContent));
}
if max_total_object_size.is_some() && size.is_none() {
return Err(S3Error::new(S3ErrorCode::UnexpectedContent));
}
if let (Some(limit), Some(size)) = (max_total_object_size, size)
&& u64::try_from(size).is_ok_and(|size| size > limit)
{
return Err(S3Error::new(S3ErrorCode::EntityTooLarge));
}
if max_total_object_size.is_some() {
let request_id = req
.extensions
.get::<super::storage_api::multipart_usecase::request_context::RequestContext>()
.map(|ctx| ctx.request_id.clone())
.unwrap_or_default();
body_stream = guard_put_object_body_read_timeout(
body_stream,
&bucket,
&key,
&request_id,
content_length,
put_object_body_read_timeout().max(Duration::from_secs(rustfs_config::DEFAULT_HTTP_REQUEST_BODY_READ_TIMEOUT)),
);
}
if size.is_none() {
let mut total = 0i64;
@@ -1106,6 +1026,16 @@ impl DefaultMultipartUsecase {
body_stream = StreamingBlob::wrap(stream);
}
// Get multipart info early to check if managed encryption will be applied
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let fi = store
.get_multipart_info(&bucket, &key, &upload_id, &opts)
.await
.map_err(ApiError::from)?;
let mut size = size.ok_or_else(|| s3_error!(UnexpectedContent))?;
let ingress_stage_start = rustfs_io_metrics::put_stage_metrics_enabled().then(std::time::Instant::now);
@@ -1320,11 +1250,6 @@ impl DefaultMultipartUsecase {
&self,
req: S3Request<ListMultipartUploadsInput>,
) -> S3Result<S3Response<ListMultipartUploadsOutput>> {
reject_presigned_multipart_max_total_object_size_for_other_operation(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedSigV4Request>().is_some(),
)?;
reject_presigned_put_max_content_length_for_other_operation(
&req.headers,
req.uri.query(),
@@ -1377,11 +1302,6 @@ impl DefaultMultipartUsecase {
}
pub async fn execute_list_parts(&self, req: S3Request<ListPartsInput>) -> S3Result<S3Response<ListPartsOutput>> {
reject_presigned_multipart_max_total_object_size_for_other_operation(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedSigV4Request>().is_some(),
)?;
reject_presigned_put_max_content_length_for_other_operation(
&req.headers,
req.uri.query(),
@@ -1418,11 +1338,6 @@ impl DefaultMultipartUsecase {
&self,
req: S3Request<UploadPartCopyInput>,
) -> S3Result<S3Response<UploadPartCopyOutput>> {
reject_presigned_multipart_max_total_object_size_for_other_operation(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedSigV4Request>().is_some(),
)?;
reject_presigned_put_max_content_length_for_other_operation(
&req.headers,
req.uri.query(),
@@ -1526,7 +1441,6 @@ impl DefaultMultipartUsecase {
.get_multipart_info(&bucket, &key, &upload_id, &dst_opts)
.await
.map_err(ApiError::from)?;
let destination_size_limit = multipart_max_total_object_size(&mp_info.user_defined)?;
EncryptionRequest {
bucket: &bucket,
key: &key,
@@ -1609,25 +1523,19 @@ impl DefaultMultipartUsecase {
return Err(s3_error!(PreconditionFailed));
}
let source_logical_size = match src_info.get_actual_size() {
Ok(size) if size >= 0 => size,
Ok(_) | Err(_) if destination_size_limit.is_some() => {
return Err(S3Error::new(S3ErrorCode::UnexpectedContent));
}
Ok(_) | Err(_) => src_info.size,
};
let (_start_offset, length) = if let Some(ref range_spec) = rs {
// Copy-source ranges are expressed over the logical plaintext object.
// Encrypted (and compressed) objects have a larger or smaller physical
// representation, so validating against `size` rejects valid later parts.
validate_copy_source_range_not_exceeds(range_spec, source_logical_size)?;
let validation_size = src_info.get_actual_size().unwrap_or(src_info.size);
validate_copy_source_range_not_exceeds(range_spec, validation_size)?;
range_spec
.get_offset_length(source_logical_size)
.get_offset_length(validation_size)
.map_err(|e| S3Error::with_message(S3ErrorCode::InvalidRange, e.to_string()))?
} else {
(0, source_logical_size)
(0, src_info.size)
};
let is_disk_compressed =
@@ -2229,16 +2137,6 @@ mod tests {
assert_eq!(metadata.get(AMZ_OBJECT_TAGGING), Some(&"project=rustfs".to_string()));
}
#[test]
fn multipart_max_total_object_size_reads_compatible_internal_metadata() {
let mut metadata = HashMap::new();
insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "104857600".to_string());
assert_eq!(multipart_max_total_object_size(&metadata).unwrap(), Some(104_857_600));
metadata.insert("x-minio-internal-max-total-object-size".to_string(), "1".to_string());
assert!(multipart_max_total_object_size(&metadata).is_err());
}
#[tokio::test]
async fn execute_complete_multipart_upload_rejects_missing_parts_payload() {
let input = CompleteMultipartUploadInput::builder()
-1
View File
@@ -195,7 +195,6 @@ pub(crate) use self::delete::*;
pub(crate) use self::extract::*;
pub(crate) use self::get::*;
use self::put::*;
pub(crate) use self::put::{guard_put_object_body_read_timeout, put_object_body_read_timeout};
pub(crate) use self::shared::*;
#[cfg(test)]
use self::test_support::*;
+3 -3
View File
@@ -90,7 +90,7 @@ fn resolve_put_object_authoritative_size(headers: &HeaderMap, content_length: Op
/// Returns `Duration::ZERO` when disabled (`RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0`),
/// in which case [`guard_put_object_body_read_timeout`] passes the body through
/// untouched.
pub(crate) fn put_object_body_read_timeout() -> Duration {
fn put_object_body_read_timeout() -> Duration {
Duration::from_secs(rustfs_utils::get_env_u64(
rustfs_config::ENV_HTTP_REQUEST_BODY_READ_TIMEOUT,
rustfs_config::DEFAULT_HTTP_REQUEST_BODY_READ_TIMEOUT,
@@ -260,7 +260,7 @@ impl ByteStream for RequestBodyReadTimeout {
/// Wrap an incoming request body with [`RequestBodyReadTimeout`] unless the
/// feature is disabled (`timeout == 0`), in which case the body is returned
/// untouched. `remaining_length` is preserved via [`StreamingBlob::new`].
pub(crate) fn guard_put_object_body_read_timeout(
fn guard_put_object_body_read_timeout(
body: StreamingBlob,
bucket: &str,
key: &str,
@@ -962,7 +962,7 @@ impl DefaultObjectUsecase {
rustfs_io_metrics::record_put_object_stage_duration_from("app_bucket_validate", bucket_validate_stage_start);
let put_admission = match get_concurrency_manager()
.admit_put_object(size)
.admit_put_object()
.await
.map_err(|_| s3_error!(InternalError, "foreground write admission closed"))?
{
-136
View File
@@ -53,7 +53,6 @@ const EVENT_SESSION_TOKEN_EXTRACTION: &str = "session_token_extraction";
/// RustFS-specific query capability for a single presigned PutObject request.
pub(crate) const RUSTFS_MAX_CONTENT_LENGTH_QUERY: &str = "x-rustfs-max-content-length";
pub(crate) const RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY: &str = "x-rustfs-max-total-object-size";
/// Inserted by the S3 access boundary after the upstream verifier accepts a
/// request as SigV4 presigned. Downstream capability parsing must require this
@@ -61,9 +60,6 @@ pub(crate) const RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY: &str = "x-rustfs-max-total-
#[derive(Debug, Clone, Copy)]
pub(crate) struct VerifiedPresignedRequest;
#[derive(Debug, Clone, Copy)]
pub(crate) struct VerifiedSigV4Request;
/// Performs constant-time string comparison to prevent timing attacks.
///
/// This function should be used when comparing sensitive values like passwords,
@@ -1115,92 +1111,6 @@ pub(crate) fn reject_presigned_put_max_content_length_for_other_operation(
Ok(())
}
/// Parse the V2 multipart total-size capability after SigV4 authentication.
/// Header-authenticated CreateMultipartUpload requests are accepted because the
/// custom query is covered by the SigV4 canonical request; later multipart
/// operations read the immutable value from the upload session metadata.
pub(crate) fn parse_presigned_multipart_max_total_object_size(
header: &HeaderMap,
query: Option<&str>,
verified_sigv4: bool,
) -> S3Result<Option<u64>> {
let Some(query) = query else {
return Ok(None);
};
let mut value = None;
let mut decoded_query = Vec::new();
for (name, candidate) in form_urlencoded::parse(query.as_bytes()) {
decoded_query.push((name.to_string(), candidate.to_string()));
if name == RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY {
if value.is_some() {
return Err(S3Error::with_message(
S3ErrorCode::InvalidRequest,
format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} must appear exactly once"),
));
}
value = Some(candidate.into_owned());
} else if name.eq_ignore_ascii_case(RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY) {
return Err(S3Error::with_message(
S3ErrorCode::InvalidRequest,
format!("query parameter name must be exactly {RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY}"),
));
}
}
let Some(value) = value else {
return Ok(None);
};
let auth_type = get_request_auth_type_with_query(header, Some(query));
let is_presigned = matches!(auth_type, AuthType::Presigned);
let is_header_signed = matches!(auth_type, AuthType::Signed);
let complete_presigned_query = [
("x-amz-algorithm", "AWS4-HMAC-SHA256"),
("x-amz-date", ""),
("x-amz-expires", ""),
("x-amz-signedheaders", ""),
("x-amz-credential", ""),
("x-amz-signature", ""),
]
.into_iter()
.all(|(name, expected)| {
decoded_query
.iter()
.find(|(candidate, _)| candidate.eq_ignore_ascii_case(name))
.is_some_and(|(_, candidate)| !candidate.is_empty() && (expected.is_empty() || candidate == expected))
});
let authenticated = verified_sigv4 && (is_header_signed || (is_presigned && complete_presigned_query));
if !authenticated {
return Err(S3Error::with_message(
S3ErrorCode::InvalidRequest,
format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} requires a verified SigV4 request"),
));
}
value.parse::<u64>().map(Some).map_err(|_| {
S3Error::with_message(
S3ErrorCode::InvalidRequest,
format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} must be a non-negative 64-bit integer"),
)
})
}
pub(crate) fn reject_presigned_multipart_max_total_object_size_for_other_operation(
header: &HeaderMap,
query: Option<&str>,
verified_sigv4: bool,
) -> S3Result<()> {
if parse_presigned_multipart_max_total_object_size(header, query, verified_sigv4)?.is_some() {
return Err(S3Error::with_message(
S3ErrorCode::InvalidRequest,
format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} is only supported for CreateMultipartUpload"),
));
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
@@ -1906,52 +1816,6 @@ mod tests {
}
}
#[test]
fn multipart_max_total_object_size_requires_signed_create_request() {
let headers = HeaderMap::new();
let signed_prefix = "X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20260827T000000Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Credential=test/20260827/us-east-1/s3/aws4_request&X-Amz-Signature=signature";
let query = format!("{signed_prefix}&x-rustfs-max-total-object-size=104857600");
assert_eq!(
parse_presigned_multipart_max_total_object_size(&headers, Some(&query), true).unwrap(),
Some(104_857_600)
);
assert_eq!(
reject_presigned_multipart_max_total_object_size_for_other_operation(&headers, Some(&query), true)
.unwrap_err()
.code(),
&S3ErrorCode::InvalidRequest
);
}
#[test]
fn multipart_max_total_object_size_rejects_tampering_and_invalid_values() {
let headers = HeaderMap::new();
let signed_prefix = "X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20260827T000000Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Credential=test/20260827/us-east-1/s3/aws4_request&X-Amz-Signature=signature";
for query in [
"x-rustfs-max-total-object-size=1",
"X-RustFS-Max-Total-Object-Size=1",
"X-Amz-Algorithm=AWS4-HMAC-SHA256&x-rustfs-max-total-object-size=1&x-rustfs-max-total-object-size=2",
"X-Amz-Algorithm=AWS4-HMAC-SHA256&x-rustfs-max-total-object-size=-1",
"X-Amz-Algorithm=AWS4-HMAC-SHA256&x-rustfs-max-total-object-size=18446744073709551616",
] {
assert_eq!(
parse_presigned_multipart_max_total_object_size(&headers, Some(query), true)
.unwrap_err()
.code(),
&S3ErrorCode::InvalidRequest
);
}
let forged = format!("{signed_prefix}&x-rustfs-max-total-object-size=1");
assert_eq!(
parse_presigned_multipart_max_total_object_size(&headers, Some(&forged), false)
.unwrap_err()
.code(),
&S3ErrorCode::InvalidRequest
);
}
#[test]
fn test_credentials_is_expired() {
let mut cred = create_test_credentials();
-1
View File
@@ -372,7 +372,6 @@ impl From<StorageError> for ApiError {
StorageError::ObjectExistsAsDirectory(_, _) => S3ErrorCode::InvalidArgument,
StorageError::InvalidPart(_, _, _) => S3ErrorCode::InvalidPart,
StorageError::EntityTooSmall(_, _, _) => S3ErrorCode::EntityTooSmall,
StorageError::EntityTooLarge(_, _) => S3ErrorCode::EntityTooLarge,
StorageError::PreconditionFailed => S3ErrorCode::PreconditionFailed,
StorageError::NotModified => S3ErrorCode::NotModified,
StorageError::InvalidRangeSpec(_) => S3ErrorCode::InvalidRange,
+4 -18
View File
@@ -16,10 +16,9 @@ use super::ObjectOptions;
use super::ecfs::FS;
use super::{ECStore, PolicySys, ReplicationStatusType, StorageError, get_lock_acquire_timeout, is_err_bucket_not_found};
use crate::auth::{
AuthType, RUSTFS_MAX_CONTENT_LENGTH_QUERY, RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY, VerifiedPresignedRequest,
VerifiedSigV4Request, check_key_valid_with_context, get_condition_values_with_client_info,
get_condition_values_with_query_and_client_info, get_request_auth_type_with_query, get_session_token,
parse_presigned_multipart_max_total_object_size, parse_presigned_put_max_content_length,
AuthType, RUSTFS_MAX_CONTENT_LENGTH_QUERY, VerifiedPresignedRequest, check_key_valid_with_context,
get_condition_values_with_client_info, get_condition_values_with_query_and_client_info, get_request_auth_type_with_query,
get_session_token, parse_presigned_put_max_content_length,
};
use crate::error::ApiError;
use crate::license::license_check;
@@ -1772,9 +1771,7 @@ impl S3Access for FS {
// Publish this server's context slot so downstream data-plane handlers
// resolve the same store (backlog#1052 S6).
let auth_type = get_request_auth_type_with_query(cx.headers(), cx.uri().query());
let verified_presigned = matches!(auth_type, AuthType::Presigned);
let verified_sigv4 = matches!(auth_type, AuthType::Presigned | AuthType::Signed);
let verified_presigned = matches!(get_request_auth_type_with_query(cx.headers(), cx.uri().query()), AuthType::Presigned);
{
let ext = cx.extensions_mut();
ext.insert(self.server_ctx().clone());
@@ -1782,9 +1779,6 @@ impl S3Access for FS {
if verified_presigned {
ext.insert(VerifiedPresignedRequest);
}
if verified_sigv4 {
ext.insert(VerifiedSigV4Request);
}
}
// The size capability is intentionally scoped to the single-object
@@ -1799,14 +1793,6 @@ impl S3Access for FS {
format!("{RUSTFS_MAX_CONTENT_LENGTH_QUERY} is only supported for presigned PutObject"),
));
}
if parse_presigned_multipart_max_total_object_size(cx.headers(), cx.uri().query(), verified_sigv4)?.is_some()
&& cx.s3_op().name() != "CreateMultipartUpload"
{
return Err(S3Error::with_message(
S3ErrorCode::InvalidRequest,
format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} is only supported for CreateMultipartUpload"),
));
}
license_check().map_err(|er| match er.kind() {
std::io::ErrorKind::PermissionDenied => s3_error!(AccessDenied, "{er}"),
_ => {
+89 -330
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Concurrency manager for coordinating concurrent GetObject and PutObject requests.
//! Concurrency manager for coordinating concurrent GetObject requests.
use super::io_schedule::{
IoLoadLevel, IoLoadMetrics, IoPriority, IoPriorityQueue, IoPriorityQueueConfig, IoQueueStatus, IoSchedulerConfig, IoStrategy,
@@ -34,8 +34,6 @@ use std::time::Duration;
use tokio::sync::Semaphore;
use tracing::debug;
const DERIVED_LARGE_PUT_ADMISSION_LIMIT_MAX: usize = 32;
/// Global concurrency manager instance
pub(crate) static CONCURRENCY_MANAGER: LazyLock<ConcurrencyManager> = LazyLock::new(ConcurrencyManager::new);
@@ -67,8 +65,11 @@ pub struct ConcurrencyManager {
bandwidth_monitor: Arc<Mutex<BandwidthMonitor>>,
/// Metrics collector for I/O latency tracking (P50, P95, P99)
metrics_collector: Arc<MetricsCollector>,
/// Foreground PutObject admission policy, resolved once at startup.
put_admission_policy: PutAdmissionPolicy,
/// Experimental fixed-count foreground PutObject admission gate.
put_admission_semaphore: Arc<Semaphore>,
put_admission_enabled: bool,
put_admission_limit: usize,
put_admission_wait_timeout: Duration,
}
impl std::fmt::Debug for ConcurrencyManager {
@@ -126,201 +127,10 @@ pub enum PutObjectAdmission {
/// Request is admitted and must hold the permit until the store write
/// returns or the request fails before mutation.
Admitted(tokio::sync::OwnedSemaphorePermit),
/// The selected foreground PUT admission gate stayed full until the configured wait timeout.
/// The fixed-count gate stayed full until the configured wait timeout.
Rejected,
}
#[derive(Clone)]
struct PutAdmissionGate {
semaphore: Arc<Semaphore>,
limit: usize,
wait_timeout: Duration,
}
impl PutAdmissionGate {
fn new(limit: usize, wait_timeout: Duration) -> Self {
Self {
semaphore: Arc::new(Semaphore::new(limit)),
limit,
wait_timeout,
}
}
fn active(&self) -> usize {
self.limit.saturating_sub(self.semaphore.available_permits())
}
async fn admit(&self) -> Result<PutObjectAdmission, tokio::sync::AcquireError> {
if self.wait_timeout.is_zero() {
return Ok(match self.semaphore.clone().try_acquire_owned() {
Ok(permit) => PutObjectAdmission::Admitted(permit),
Err(tokio::sync::TryAcquireError::NoPermits) => PutObjectAdmission::Rejected,
Err(tokio::sync::TryAcquireError::Closed) => PutObjectAdmission::Rejected,
});
}
match tokio::time::timeout(self.wait_timeout, self.semaphore.clone().acquire_owned()).await {
Ok(permit) => Ok(PutObjectAdmission::Admitted(permit?)),
Err(_) => Ok(PutObjectAdmission::Rejected),
}
}
}
#[derive(Clone)]
enum PutAdmissionPolicy {
/// Strict admission was explicitly enabled with limit `0`.
Disabled,
/// No hard PUT gate is configured; foreground write snapshots use the
/// existing active request counter as a soft pressure signal.
LegacyCounterOnly,
/// Explicit all-PUT admission gate.
Strict(PutAdmissionGate),
/// Default large/unknown-size PUT admission gate.
Large { gate: PutAdmissionGate, min_size_bytes: usize },
}
impl PutAdmissionPolicy {
fn from_env(max_disk_reads: usize) -> Self {
let strict_enabled = rustfs_utils::get_env_bool(
rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_ENABLE,
rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE,
);
if strict_enabled {
let strict_limit = rustfs_utils::get_env_usize(
rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_LIMIT,
rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_LIMIT,
);
let strict_wait_timeout = Duration::from_millis(rustfs_utils::get_env_u64(
rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
));
return if strict_limit == 0 {
Self::Disabled
} else {
Self::Strict(PutAdmissionGate::new(strict_limit, strict_wait_timeout))
};
}
let large_enabled = rustfs_utils::get_env_bool(
rustfs_config::ENV_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE,
rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE,
);
if !large_enabled {
return Self::LegacyCounterOnly;
}
let large_limit = derive_large_put_admission_limit(
rustfs_utils::get_env_usize(
rustfs_config::ENV_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT,
rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT,
),
max_disk_reads,
);
let min_size_bytes = rustfs_utils::get_env_usize(
rustfs_config::ENV_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES,
rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES,
);
let wait_timeout = Duration::from_millis(rustfs_utils::get_env_u64(
rustfs_config::ENV_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
));
Self::Large {
gate: PutAdmissionGate::new(large_limit, wait_timeout),
min_size_bytes,
}
}
#[cfg(test)]
fn strict_for_test(enabled: bool, limit: usize, wait_timeout: Duration) -> Self {
if enabled {
if limit == 0 {
Self::Disabled
} else {
Self::Strict(PutAdmissionGate::new(limit, wait_timeout))
}
} else {
Self::LegacyCounterOnly
}
}
#[cfg(test)]
fn large_for_test(enabled: bool, limit: usize, min_size_bytes: usize, wait_timeout: Duration) -> Self {
if enabled && limit > 0 {
Self::Large {
gate: PutAdmissionGate::new(limit, wait_timeout),
min_size_bytes,
}
} else {
Self::LegacyCounterOnly
}
}
async fn admit(&self, size: i64) -> Result<PutObjectAdmission, tokio::sync::AcquireError> {
match self {
Self::Disabled | Self::LegacyCounterOnly => Ok(PutObjectAdmission::Disabled),
Self::Strict(gate) => gate.admit().await,
Self::Large { gate, min_size_bytes } if should_gate_large_put(size, *min_size_bytes) => gate.admit().await,
Self::Large { .. } => Ok(PutObjectAdmission::Disabled),
}
}
fn snapshot(&self, legacy_limit: usize) -> WorkloadAdmissionSnapshot {
match self {
Self::Disabled => put_admission_snapshot(0, 0, None),
Self::LegacyCounterOnly => put_admission_snapshot(PutObjectGuard::concurrent_count(), legacy_limit, None),
Self::Strict(gate) => {
put_admission_snapshot(gate.active(), gate.limit, Some("foreground write admission permits exhausted"))
}
Self::Large { gate, .. } => {
put_admission_snapshot(gate.active(), gate.limit, Some("large foreground write admission permits exhausted"))
}
}
}
}
fn put_admission_snapshot(active: usize, limit: usize, hard_gate_reason: Option<&'static str>) -> WorkloadAdmissionSnapshot {
let state = if limit == 0 {
AdmissionState::Disabled
} else if active >= limit {
AdmissionState::Saturated
} else {
AdmissionState::Open
};
let admission =
WorkloadAdmissionSnapshot::new(WorkloadClass::ForegroundWrite, state).with_counts(Some(active), None, Some(limit));
match state {
AdmissionState::Disabled => admission.with_reason("foreground write admission disabled"),
AdmissionState::Saturated => {
admission.with_reason(hard_gate_reason.unwrap_or("foreground write concurrency reached local pressure limit"))
}
_ => admission,
}
}
fn derive_large_put_admission_limit(configured_limit: usize, max_disk_reads: usize) -> usize {
if configured_limit > 0 {
return configured_limit;
}
let scheduler_base = if max_disk_reads == 0 {
rustfs_config::DEFAULT_OBJECT_MAX_CONCURRENT_DISK_READS
} else {
max_disk_reads
};
scheduler_base.div_ceil(2).clamp(1, DERIVED_LARGE_PUT_ADMISSION_LIMIT_MAX)
}
fn should_gate_large_put(size: i64, min_size_bytes: usize) -> bool {
if min_size_bytes == 0 || size < 0 {
return true;
}
usize::try_from(size).is_ok_and(|size| size >= min_size_bytes)
}
impl ConcurrencyManager {
/// Create a new concurrency manager with default settings
///
@@ -367,7 +177,18 @@ impl ConcurrencyManager {
// Initialize metrics collector for I/O latency tracking
// Keep 1000 samples for P95/P99 calculation
let metrics_collector = Arc::new(MetricsCollector::new(performance_metrics, 1000));
let put_admission_policy = PutAdmissionPolicy::from_env(max_disk_reads);
let put_admission_enabled = rustfs_utils::get_env_bool(
rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_ENABLE,
rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE,
);
let put_admission_limit = rustfs_utils::get_env_usize(
rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_LIMIT,
rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_LIMIT,
);
let put_admission_wait_timeout = Duration::from_millis(rustfs_utils::get_env_u64(
rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
));
// Build queue config directly from scheduler config.
let queue_config = IoPriorityQueueConfig::from_scheduler_config(&scheduler_config);
@@ -383,7 +204,10 @@ impl ConcurrencyManager {
pattern_detector,
bandwidth_monitor,
metrics_collector,
put_admission_policy,
put_admission_semaphore: Arc::new(Semaphore::new(if put_admission_enabled { put_admission_limit } else { 0 })),
put_admission_enabled,
put_admission_limit,
put_admission_wait_timeout,
}
}
@@ -410,19 +234,10 @@ impl ConcurrencyManager {
#[cfg(test)]
pub(crate) fn with_put_admission_for_test(enabled: bool, limit: usize, wait_timeout: Duration) -> Self {
let mut manager = Self::new();
manager.put_admission_policy = PutAdmissionPolicy::strict_for_test(enabled, limit, wait_timeout);
manager
}
#[cfg(test)]
pub(crate) fn with_large_put_admission_for_test(
enabled: bool,
limit: usize,
min_size_bytes: usize,
wait_timeout: Duration,
) -> Self {
let mut manager = Self::new();
manager.put_admission_policy = PutAdmissionPolicy::large_for_test(enabled, limit, min_size_bytes, wait_timeout);
manager.put_admission_semaphore = Arc::new(Semaphore::new(if enabled { limit } else { 0 }));
manager.put_admission_enabled = enabled;
manager.put_admission_limit = limit;
manager.put_admission_wait_timeout = wait_timeout;
manager
}
@@ -511,13 +326,30 @@ impl ConcurrencyManager {
}
}
/// Admit a foreground PutObject request under the configured write gate.
/// Admit a foreground PutObject request under the experimental fixed-count gate.
///
/// The strict experimental gate applies to every PUT only when explicitly
/// enabled. Otherwise the default-on large-object gate protects sustained
/// erasure/RPC pressure while keeping small PUTs on the legacy path.
pub async fn admit_put_object(&self, size: i64) -> Result<PutObjectAdmission, tokio::sync::AcquireError> {
self.put_admission_policy.admit(size).await
/// The default-off path returns [`PutObjectAdmission::Disabled`] without
/// touching the semaphore, preserving legacy behavior. When enabled, the
/// permit must be acquired before body ingest and held until the store write
/// returns, so saturated foreground writes can fail with `SlowDown` before
/// creating visible side effects.
pub async fn admit_put_object(&self) -> Result<PutObjectAdmission, tokio::sync::AcquireError> {
if !self.put_admission_enabled || self.put_admission_limit == 0 {
return Ok(PutObjectAdmission::Disabled);
}
if self.put_admission_wait_timeout.is_zero() {
return Ok(match self.put_admission_semaphore.clone().try_acquire_owned() {
Ok(permit) => PutObjectAdmission::Admitted(permit),
Err(tokio::sync::TryAcquireError::NoPermits) => PutObjectAdmission::Rejected,
Err(tokio::sync::TryAcquireError::Closed) => PutObjectAdmission::Rejected,
});
}
match tokio::time::timeout(self.put_admission_wait_timeout, self.put_admission_semaphore.clone().acquire_owned()).await {
Ok(permit) => Ok(PutObjectAdmission::Admitted(permit?)),
Err(_) => Ok(PutObjectAdmission::Rejected),
}
}
// ============================================
@@ -928,7 +760,35 @@ impl ConcurrencyManager {
/// Get a read-only workload admission snapshot for foreground writes.
pub fn put_object_admission_snapshot(&self) -> WorkloadAdmissionSnapshot {
self.put_admission_policy.snapshot(self.scheduler_config.max_concurrent_reads)
let (active, limit, hard_gate_enabled) = if self.put_admission_enabled && self.put_admission_limit > 0 {
(
self.put_admission_limit
.saturating_sub(self.put_admission_semaphore.available_permits()),
self.put_admission_limit,
true,
)
} else {
(PutObjectGuard::concurrent_count(), self.scheduler_config.max_concurrent_reads, false)
};
let state = if limit == 0 {
AdmissionState::Disabled
} else if active >= limit {
AdmissionState::Saturated
} else {
AdmissionState::Open
};
let admission =
WorkloadAdmissionSnapshot::new(WorkloadClass::ForegroundWrite, state).with_counts(Some(active), None, Some(limit));
match state {
AdmissionState::Disabled => admission.with_reason("foreground write admission disabled"),
AdmissionState::Saturated if hard_gate_enabled => {
admission.with_reason("foreground write admission permits exhausted")
}
AdmissionState::Saturated => admission.with_reason("foreground write concurrency reached local pressure limit"),
_ => admission,
}
}
/// Get a read-only workload admission registry snapshot for local storage concurrency.
@@ -1002,7 +862,7 @@ impl Default for ConcurrencyManager {
mod integration_tests {
use super::super::io_schedule::{IoLoadLevel, IoPriority};
use super::super::request_guard::GetObjectGuard;
use super::{ConcurrencyManager, PutObjectAdmission, derive_large_put_admission_limit};
use super::{ConcurrencyManager, PutObjectAdmission};
use crate::storage::storage_api::concurrency_consumer::PutObjectGuard;
use rustfs_concurrency::{AdmissionState, WorkloadAdmissionSnapshotProvider, WorkloadClass};
use rustfs_io_core::io_profile::{AccessPattern, StorageMedia};
@@ -1081,7 +941,7 @@ mod integration_tests {
#[serial]
async fn test_concurrency_manager_workload_admission_snapshot_tracks_put_requests() {
crate::storage::concurrency::reset_active_put_requests();
let manager = ConcurrencyManager::with_put_admission_for_test(false, 0, Duration::ZERO);
let manager = ConcurrencyManager::new();
let initial = manager.put_object_admission_snapshot();
assert_eq!(initial.class, WorkloadClass::ForegroundWrite);
@@ -1105,42 +965,26 @@ mod integration_tests {
let manager = ConcurrencyManager::with_put_admission_for_test(false, 1, Duration::ZERO);
let admission = manager
.admit_put_object(1024)
.admit_put_object()
.await
.expect("disabled put admission must not close");
assert!(matches!(admission, PutObjectAdmission::Disabled));
assert_eq!(manager.put_admission_semaphore.available_permits(), 0);
assert_eq!(manager.put_object_admission_snapshot().state, AdmissionState::Open);
}
#[tokio::test]
#[serial]
async fn test_concurrency_manager_strict_put_admission_zero_limit_disables_large_gate() {
let manager = ConcurrencyManager::with_put_admission_for_test(true, 0, Duration::ZERO);
let admission = manager
.admit_put_object(32 * 1024 * 1024)
.await
.expect("strict zero-limit put admission must not close");
assert!(matches!(admission, PutObjectAdmission::Disabled));
assert_eq!(manager.put_object_admission_snapshot().state, AdmissionState::Disabled);
}
#[tokio::test]
#[serial]
async fn test_concurrency_manager_put_admission_rejects_when_limit_full() {
let manager = ConcurrencyManager::with_put_admission_for_test(true, 1, Duration::ZERO);
let first = manager
.admit_put_object(1024)
.await
.expect("first put admission should acquire");
let first = manager.admit_put_object().await.expect("first put admission should acquire");
assert!(matches!(first, PutObjectAdmission::Admitted(_)));
assert_eq!(manager.put_object_admission_snapshot().state, AdmissionState::Saturated);
let second = manager
.admit_put_object(1024)
.admit_put_object()
.await
.expect("full put admission gate should reject, not close");
assert!(matches!(second, PutObjectAdmission::Rejected));
@@ -1151,14 +995,11 @@ mod integration_tests {
async fn test_concurrency_manager_put_admission_reuses_released_permit() {
let manager = ConcurrencyManager::with_put_admission_for_test(true, 1, Duration::ZERO);
let first = manager
.admit_put_object(1024)
.await
.expect("first put admission should acquire");
let first = manager.admit_put_object().await.expect("first put admission should acquire");
drop(first);
let second = manager
.admit_put_object(1024)
.admit_put_object()
.await
.expect("released put admission permit should be reusable");
assert!(matches!(second, PutObjectAdmission::Admitted(_)));
@@ -1168,13 +1009,10 @@ mod integration_tests {
#[serial]
async fn test_concurrency_manager_put_admission_wait_timeout_rejects() {
let manager = ConcurrencyManager::with_put_admission_for_test(true, 1, Duration::from_secs(5));
let held = manager
.admit_put_object(1024)
.await
.expect("first put admission should acquire");
let held = manager.admit_put_object().await.expect("first put admission should acquire");
let waiter_manager = manager.clone();
let waiter = tokio::spawn(async move { waiter_manager.admit_put_object(1024).await });
let waiter = tokio::spawn(async move { waiter_manager.admit_put_object().await });
tokio::task::yield_now().await;
tokio::time::advance(Duration::from_secs(5)).await;
@@ -1186,85 +1024,6 @@ mod integration_tests {
drop(held);
}
#[tokio::test]
#[serial]
async fn test_concurrency_manager_large_put_admission_bypasses_small_puts() {
let min_size = rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES;
let manager = ConcurrencyManager::with_large_put_admission_for_test(true, 1, min_size, Duration::ZERO);
let held = manager
.admit_put_object(min_size as i64)
.await
.expect("large put admission should acquire");
assert!(matches!(held, PutObjectAdmission::Admitted(_)));
let small = manager
.admit_put_object((min_size - 1) as i64)
.await
.expect("small put should bypass large admission");
assert!(matches!(small, PutObjectAdmission::Disabled));
let large = manager
.admit_put_object(min_size as i64)
.await
.expect("second large put should reject when the gate is full");
assert!(matches!(large, PutObjectAdmission::Rejected));
}
#[tokio::test]
#[serial]
async fn test_concurrency_manager_large_put_admission_gates_unknown_size() {
let manager = ConcurrencyManager::with_large_put_admission_for_test(true, 1, 32 * 1024 * 1024, Duration::ZERO);
let held = manager
.admit_put_object(-1)
.await
.expect("unknown-size put admission should acquire");
assert!(matches!(held, PutObjectAdmission::Admitted(_)));
let second = manager
.admit_put_object(-1)
.await
.expect("unknown-size put admission should reject when the gate is full");
assert!(matches!(second, PutObjectAdmission::Rejected));
}
#[tokio::test]
#[serial]
async fn test_concurrency_manager_large_put_snapshot_tracks_gate() {
let manager = ConcurrencyManager::with_large_put_admission_for_test(true, 2, 32 * 1024 * 1024, Duration::ZERO);
let first = manager
.admit_put_object(32 * 1024 * 1024)
.await
.expect("first large put admission should acquire");
let initial = manager.put_object_admission_snapshot();
assert_eq!(initial.class, WorkloadClass::ForegroundWrite);
assert_eq!(initial.state, AdmissionState::Open);
assert_eq!(initial.active, Some(1));
assert_eq!(initial.limit, Some(2));
let second = manager
.admit_put_object(32 * 1024 * 1024)
.await
.expect("second large put admission should acquire");
let saturated = manager.put_object_admission_snapshot();
assert_eq!(saturated.state, AdmissionState::Saturated);
assert_eq!(saturated.active, Some(2));
assert_eq!(saturated.limit, Some(2));
drop((first, second));
}
#[test]
fn test_concurrency_manager_derives_large_put_admission_limit_from_scheduler_cap() {
assert_eq!(derive_large_put_admission_limit(7, 64), 7);
assert_eq!(derive_large_put_admission_limit(0, 64), 32);
assert_eq!(derive_large_put_admission_limit(0, 8), 4);
assert_eq!(derive_large_put_admission_limit(0, 1), 1);
assert_eq!(derive_large_put_admission_limit(0, 0), 32);
}
#[tokio::test]
#[serial]
async fn test_concurrency_manager_workload_admission_registry_covers_required_classes() {
+4 -23
View File
@@ -961,9 +961,9 @@ fn sse_kms_key_policy_enforced(principal: Option<&SseKmsPrincipal>) -> bool {
/// Report the configured SSE-KMS authorization mode once, at startup.
///
/// The disabled case warns rather than logs: while enforcement is off, any identity
/// allowed to write an object can encrypt it under any key, and operators should hear
/// about that even though disabled is the long-term default.
/// The disabled case warns rather than logs: it is the compatibility default for this
/// release only, and operators need the lead time to grant the kms actions before the
/// default flips.
pub(crate) fn log_sse_kms_key_policy_mode() {
if sse_kms_key_policy_enforced(None) {
tracing::info!(
@@ -984,7 +984,7 @@ pub(crate) fn log_sse_kms_key_policy_mode() {
"SSE-KMS requests are not authorized against the KMS key they name; any identity allowed to \
write an object may encrypt it under any key, and any identity allowed to read it may have it \
decrypted. Grant kms:GenerateDataKey and kms:Decrypt on the keys your workloads use, then set \
{ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY}=true."
{ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY}=true. A later release defaults this to enabled."
);
}
@@ -1027,25 +1027,6 @@ async fn authorize_sse_kms_key(
"Principal is not authorized for the KMS key resolved for this request"
);
// One warn per process, not per request: anonymous denials are driven by
// unauthenticated traffic, so a per-request warn would let anyone flood the
// log. Per-request detail stays on the audit entry and the debug event above.
if principal.account.is_empty() {
static ANONYMOUS_DENIAL_WARNED: std::sync::Once = std::sync::Once::new();
ANONYMOUS_DENIAL_WARNED.call_once(|| {
tracing::warn!(
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_SSE,
event = "sse_kms_anonymous_key_authorization_denied",
action = ?action,
"Anonymous requests are being denied by SSE-KMS per-key authorization: anonymous \
callers hold no kms grants, so a public bucket serving SSE-KMS objects is \
incompatible with {ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY}=true. Reported once per \
process; per-request denials are on audit entries and at debug level."
);
});
}
Err(ApiError {
code: S3ErrorCode::AccessDenied,
message: "Access Denied".to_string(),
@@ -140,14 +140,6 @@ done
latest_guard="startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')"
require_job_if "$build_workflow" "update-latest-version" " if: $latest_guard"
require_line "$build_workflow" " needs: [ build-check, publish-release ]" "latest update must follow release publication"
# Preview releases are internal validation artifacts: once the deliverable
# release is published they are deleted, while their tags stay behind.
require_job_if "$build_workflow" "cleanup-preview-releases" " if: $latest_guard"
require_line "$build_workflow" " gh release delete \"\$preview_tag\" --yes" "preview release cleanup after publication"
require_line "$build_workflow" " | select(.tag_name | startswith(\$tag + \"-preview.\"))" "cleanup must match the target's own preview tags"
require_line "$build_workflow" " | select(.tag_name | ltrimstr(\$tag + \"-preview.\") | test(\"^[0-9]+\$\"))" "cleanup must match a numeric preview iteration"
require_absent "$build_workflow" "--cleanup-tag" "preview tags must survive their release cleanup"
require_line "$build_workflow" " TARGET_COMMITISH=\$(git rev-parse --verify \"refs/tags/\${TAG}^{commit}\")" "release target commit resolution"
require_line "$build_workflow" " ./scripts/release/create_or_update_release.sh \\" "managed release creation"
require_absent "$build_workflow" "git tag -l --format='%(contents)'" "annotated tag messages must not become release notes"
-104
View File
@@ -1,104 +0,0 @@
# RustFS Heal Test
Node-outage heal test driven by
[`scripts/test/rustfs_heal_test.sh`](rustfs_heal_test.sh), based on the
Obsidian note "RustFS Heal 测试步骤". Uses the same 3-node test environment as
the pool expansion test (`vm000 vm001 vm002`).
All status checks talk to the RustFS admin API directly (SigV4-signed,
`jq` assertions), no `rc` required.
## What it does
1. Downloads the `.deb` package on all nodes (release tag or a direct URL such
as the nightly/R2 package).
2. Installs it, writes the 3x4 config
(`http://rustfs-node{1...3}:9000/data/rustfs{1...4}/mnmd`), starts all
three nodes simultaneously, verifies the cluster is up.
3. Writes data with `warp` while monitoring disk usage on the surviving nodes
(`df -B1G | grep /data/rustfs`):
- when both surviving nodes reach `STOP_NODE_AT_GB` (default 15 GiB), stop
the outage node (`vm002`, `OUTAGE_NODE_INDEX=2`);
- keep writing until both surviving nodes reach `WARP_STOP_AT_GB`
(default 40 GiB), then stop warp.
4. Restarts the outage node.
5. Starts cluster heal: `POST /rustfs/admin/v3/heal/` with body
`{"recursive":true}` (retried, returns a `clientToken`).
6. Monitors the heal task via `POST /rustfs/admin/v3/heal/?clientToken=<token>`
until the summary is a terminal success (`finished`/`completed`),
`objects_failed == 0`, **and** the outage node's disk usage reaches
`HEAL_TARGET_GB` (default 40 GiB).
7. Result analysis: heal stats (scanned/healed/failed), per-node disk usage,
pass/fail verdict.
Success requires **both** the heal API completion (the server's scan/repair
verdict) and the outage node's disk reaching the target.
## Self-hosted runner prerequisites
- Register the admin host (e.g. `heal`) as a runner with the
`smoke-testing` label.
- Install `jq`, `openssl`, `curl` and `warp` on the runner. `rc` is **not**
required.
- The runner user must be able to SSH to `vm000/vm001/vm002` without a
password prompt; nodes need passwordless `sudo` for the SSH user and
resolvable `rustfs-node*` hostnames.
- Admin API credentials need the `admin:server-info`, `admin:heal` and
`admin:rebalance` actions.
## Configuration
Same repository secrets/variables as the pool expansion workflow:
| Kind | Name | Purpose |
| ------ | --------------------- | ---------------------------------------------- |
| Secret | `RUSTFS_ACCESS_KEY` | RustFS access key (default `rustfs@test`) |
| Secret | `RUSTFS_SECRET_KEY` | RustFS secret key (default `rustfs@test`) |
| Var | `RUSTFS_API_ENDPOINT` | Admin API endpoint, e.g. `http://127.0.0.1:9000` (`RUSTFS_RC_ENDPOINT` fallback) |
| Var | `RUSTFS_NODES` | `vm000 vm001 vm002` |
| Var | `RUSTFS_SSH_USER` | `azureuser` |
| Var | `RUSTFS_NIGHTLY_PACKAGE_URL` | Default nightly deb URL (defaults to the R2 `latest` alias) |
## Workflow inputs
| Input | Default | Meaning |
| ---------------- | ------- | ----------------------------------------- |
| `package_url` | nightly | Direct `.deb` URL; empty = latest nightly |
| `stop_node_gb` | `15` | Stop outage node at N GiB on survivors |
| `warp_stop_gb` | `40` | Stop warp at N GiB on survivors |
| `heal_target_gb` | `40` | Outage node must reach N GiB after heal |
| `cleanup_before` | `true` | Reset nodes before the test |
| `cleanup_after` | `true` | Reset nodes after the test |
> ⚠️ `--reset` purges the `rustfs` package and deletes the data directories on
> all nodes. Only run against a dedicated test environment.
## Manual usage
```bash
./scripts/test/rustfs_heal_test.sh --all -y \
--package-url https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb \
--endpoint http://127.0.0.1:9000
./scripts/test/rustfs_heal_test.sh --steps 5,6,7
./scripts/test/rustfs_heal_test.sh --reset -y
```
## Known issues
- Nightly builds gate pool/rebalance activation on a live fleet capability
proof (rustfs/backlog#2031); the script retries heal/rebalance starts and
prints a hint when the signature appears.
- The cluster-level `GET /rustfs/admin/v3/background-heal/status` aggregator
returns 501 in the single-pool 3x4 topology (no notification system), so the
script monitors the started heal task via its `clientToken` instead.
- The heal task may report `progress: null` while running; the script logs
this as evidence (rustfs/backlog#2035) rather than coercing it to zero, and
reads the canonical camelCase progress fields
(`objectsScanned`/`objectsHealed`/`objectsFailed`/`progressPercentage`) with
a snake_case fallback.
- The server-side per-task heal timeout defaults to 5 minutes; the script
writes `RUSTFS_HEAL_TASK_TIMEOUT_SECS=21600` (6h) into the node config so a
multi-tens-of-GiB heal can finish. The background scanner is disabled
(`RUSTFS_HEAL_AUTO_HEAL_ENABLE=false`) so the explicit heal is the only
repair mechanism and the outage effect stays observable.
File diff suppressed because it is too large Load Diff
+17 -86
View File
@@ -93,9 +93,6 @@ WARP_BUCKET="test-10mb"
WARP_OBJ_SIZE="100MiB"
WARP_CONCURRENT=32
WARP_DURATION="5m"
# Warp log path; empty = auto-created unique temp file (the runner user may
# not be able to write a shared /tmp path owned by another user).
WARP_LOG_FILE="${RUSTFS_WARP_LOG_FILE:-}"
STORAGE_THRESHOLD=85 # stop writing when usage reaches N% (note suggests 80-85)
POLL_INTERVAL=30 # status polling interval (seconds)
@@ -105,8 +102,6 @@ DECOMMISSION_TIMEOUT=86400
SERVICE_TIMEOUT=300
DECOMMISSION_RETRIES=3 # auto clear+retry attempts after a failed decommission
DECOMMISSION_RETRY_DELAY=30 # delay between retries (seconds)
REBALANCE_START_RETRIES=6 # rebalance start retries (fleet proof may take ~10-20s after a topology change)
REBALANCE_START_RETRY_DELAY=20 # delay between rebalance start retries (seconds)
# Pool to decommission (zero-based; 0 in the note)
DECOMMISSION_POOL_ID=0
@@ -330,45 +325,9 @@ wait_service_active() {
sleep 5
waited=$((waited + 5))
done
diagnose_node_start_failure "${node}"
die "${node}: timed out waiting for ${RUSTFS_SERVICE} (${SERVICE_TIMEOUT}s)"
}
# Known server-side issues the test can hit. Format:
# "<error signature>|<tracking>|<hint>"
KNOWN_SERVER_ISSUES=(
"pool activation requires a live fleet capability proof|rustfs/backlog#2031|server-side cold-start recovery is covered by this PR; if this appears, collect node journals and treat it as a regression"
)
# Print a hint when $1 matches a known server-side issue signature.
hint_server_issue() {
local text="$1" entry sig tracking hint
for entry in "${KNOWN_SERVER_ISSUES[@]}"; do
sig="${entry%%|*}"
tracking="${entry#*|}"
hint="${tracking#*|}"
tracking="${tracking%%|*}"
if printf '%s' "${text}" | grep -qiF "${sig}"; then
printf '\033[1;33m[KNOWN SERVER ISSUE]\033[0m %s (%s): %s\n' "${sig}" "${tracking}" "${hint}" >&2
return 0
fi
done
return 1
}
# Fetch the journal tail from a node whose service failed to start and
# annotate known server-side issues.
diagnose_node_start_failure() {
local node="$1" journal
if ! journal="$(ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" \
"SUDO=\"\"; [ \"\$(id -u)\" -ne 0 ] && SUDO=\"sudo -n\"; \${SUDO} journalctl -u ${RUSTFS_SERVICE} --no-pager -n 60 2>/dev/null || true")"; then
journal="unable to collect journal (SSH command failed)"
fi
printf '%s\n' "--- ${node}: ${RUSTFS_SERVICE} journal (last 60 lines) ---" >&2
printf '%s\n' "${journal}" >&2
hint_server_issue "${journal}" || true
}
# Generate the /etc/default/rustfs content
rustfs_config_body() {
local volumes="$1"
@@ -447,13 +406,9 @@ service_action() {
local action="$1" node="$2"
log "${node}: systemctl ${action} ${RUSTFS_SERVICE}"
if [ "${DRY_RUN}" -eq 1 ]; then return 0; fi
if ! ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" \
"if [ \"\$(id -u)\" -ne 0 ]; then sudo -n systemctl ${action} ${RUSTFS_SERVICE}; else systemctl ${action} ${RUSTFS_SERVICE}; fi"; then
if [ "${action}" = "start" ]; then
diagnose_node_start_failure "${node}"
fi
die "${node}: systemctl ${action} failed"
fi
ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" \
"if [ \"\$(id -u)\" -ne 0 ]; then sudo -n systemctl ${action} ${RUSTFS_SERVICE}; else systemctl ${action} ${RUSTFS_SERVICE}; fi" \
|| die "${node}: systemctl ${action} failed"
}
service_action_all() {
@@ -632,29 +587,6 @@ wait_rebalance() {
die "timed out waiting for rebalance (${REBALANCE_TIMEOUT}s)"
}
# Start rebalance via the admin API. Nightly builds gate rebalance activation
# on a live cross-pool fence fleet capability proof that is re-established
# shortly after a pool joins, so retry a few times before failing.
start_rebalance_with_retry() {
local attempts="${REBALANCE_START_RETRIES}" delay="${REBALANCE_START_RETRY_DELAY}" attempt=1 body code id
while :; do
body="$(admin_api POST /rustfs/admin/v3/rebalance/start "")"
code="$(admin_api_code)"
if [ "${code}" = "200" ]; then
id="$(printf '%s' "${body}" | jq -r '.id // empty')"
log "rebalance started: id=${id}"
return 0
fi
warn "rebalance start attempt ${attempt}/${attempts} failed (HTTP ${code}): ${body}"
if [ "${attempt}" -ge "${attempts}" ]; then
hint_server_issue "${body}" || true
die "rebalance start failed after ${attempts} attempts (see last error above)"
fi
attempt=$((attempt + 1))
sleep "${delay}"
done
}
# Print a detailed decommission failure/progress report for one pool
print_decommission_detail() {
local body="$1" pool_id="$2" label="$3"
@@ -885,18 +817,16 @@ step4_write_data() {
log "DRY-RUN: monitoring storage usage until ${STORAGE_THRESHOLD}%"
return 0
fi
local warp_log warp_pid
warp_log="${WARP_LOG_FILE:-$(mktemp "${TMPDIR:-/tmp}/rustfs-warp.XXXXXX.log")}"
log "starting warp writes (background), log: ${warp_log}"
log "starting warp writes (background)..."
warp put --host "${API_ENDPOINT#http://}" \
--bucket "${WARP_BUCKET}" \
--access-key "${ACCESS_KEY}" \
--secret-key "${SECRET_KEY}" \
--obj.size "${WARP_OBJ_SIZE}" \
--concurrent "${WARP_CONCURRENT}" \
--noprefix --duration "${WARP_DURATION}" --noclear >"${warp_log}" 2>&1 &
warp_pid=$!
log "warp PID=${warp_pid}"
--noprefix --duration "${WARP_DURATION}" --noclear >/tmp/rustfs-warp.log 2>&1 &
local warp_pid=$!
log "warp PID=${warp_pid}, log /tmp/rustfs-warp.log"
trap 'kill "${warp_pid:-}" 2>/dev/null || true' EXIT
monitor_storage "${STORAGE_THRESHOLD}" "${warp_pid}"
kill "${warp_pid}" 2>/dev/null || true
@@ -936,8 +866,12 @@ step5_expand_pool2() {
step6_rebalance() {
log "step 6: start data rebalance (admin API)"
confirm "About to start rebalance (POST ${API_ENDPOINT}/rustfs/admin/v3/rebalance/start). Continue?"
local body id
if [ "${DRY_RUN}" -eq 0 ]; then
start_rebalance_with_retry
body="$(admin_api POST /rustfs/admin/v3/rebalance/start "")"
[ "$(admin_api_code)" = "200" ] || die "rebalance start failed (HTTP $(admin_api_code)): ${body}"
id="$(printf '%s' "${body}" | jq -r '.id // empty')"
log "rebalance started: id=${id}"
fi
wait_rebalance
}
@@ -960,8 +894,12 @@ step7_expand_pool3() {
step8_rebalance() {
log "step 8: start data rebalance (admin API)"
confirm "About to start rebalance (POST ${API_ENDPOINT}/rustfs/admin/v3/rebalance/start). Continue?"
local body id
if [ "${DRY_RUN}" -eq 0 ]; then
start_rebalance_with_retry
body="$(admin_api POST /rustfs/admin/v3/rebalance/start "")"
[ "$(admin_api_code)" = "200" ] || die "rebalance start failed (HTTP $(admin_api_code)): ${body}"
id="$(printf '%s' "${body}" | jq -r '.id // empty')"
log "rebalance started: id=${id}"
fi
wait_rebalance
}
@@ -990,7 +928,6 @@ step9_decommission() {
break
fi
if [ "${attempt}" -ge "${DECOMMISSION_RETRIES}" ]; then
warn "if the source bucket has many objects and the tested version is 1.0.0-rc.3, this is the known metacache-listing decommission bug; remove the test bucket (rc rb --force rustfs/${WARP_BUCKET}) or lower --storage-threshold, then re-run step 9"
die "pool ${DECOMMISSION_POOL_ID} still failed after ${attempt} attempts; investigate manually (POST ${API_ENDPOINT}/rustfs/admin/v3/pools/clear?by-id=true&pool=${DECOMMISSION_POOL_ID} to reset)"
fi
warn "attempt ${attempt} failed; clearing metadata and retrying in ${DECOMMISSION_RETRY_DELAY}s"
@@ -1128,12 +1065,6 @@ main() {
trap 'rm -f "${ADMIN_API_CODE_FILE}"' EXIT
if [ -n "${LOG_FILE}" ]; then
mkdir -p "$(dirname "${LOG_FILE}")"
if ! touch "${LOG_FILE}" 2>/dev/null; then
# A fixed /tmp path may be owned by another user (e.g. a previous root
# run); fall back to a unique, always-writable temp file.
LOG_FILE="$(mktemp "${TMPDIR:-/tmp}/rustfs-pool-test.XXXXXX.log")"
warn "log file not writable; using ${LOG_FILE}"
fi
exec > >(tee -a "${LOG_FILE}") 2>&1
fi
if [ "${RESET}" -eq 1 ]; then