fix(startup): avoid panic when system CA roots are missing

Co-authored-by: heihutu <[email protected]>

.agents/skills/rustfs-release-publish/SKILL.md

@@ -6,7 +6,7 @@ description: "Run the end-to-end RustFS console gate, version bump, preview vali
 
 This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
 
-Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
+Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. That Release is temporary: `build.yml` deletes it automatically once the final tag's Release is published, so the Releases page ends up carrying deliverables only while the `-preview.N` tags stay behind as the traceability record. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
 
 Pipeline shape:
 
@@ -19,6 +19,7 @@ check console main against its latest Release
   -> validate with latest rc client
   -> report preview acceptance results -> STOP for explicit human confirmation
   -> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
+  -> CI deletes the <target>-preview.N Releases (tags kept)
 ```
 
 On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
@@ -51,14 +52,16 @@ Rules:
 - Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
 - The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
 - A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
+- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
 
 ## Hard rules
 
 - Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
-- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
+- Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
+- Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
 - Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
 - The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
-- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
+- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
 - Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
 - Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
 - Completing preview acceptance does not authorize the final tag. After Phases 3–5 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
@@ -230,6 +233,7 @@ git push origin "<target>"
 - CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
 - Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
 - Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
+- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
 - Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
 
 ## Output contract
@@ -239,5 +243,5 @@ Always report:
 - Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
 - Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
 - Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
-- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
+- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
 - Any deviation from this pipeline and why the user approved it.

.github/actionlint.yaml

@@ -5,3 +5,4 @@ self-hosted-runner:
     - sm-standard-2
     - sm-standard-4
     - dind-sm-standard-2
+    - smoke-testing

.github/workflows/build.yml

@@ -1033,6 +1033,55 @@ jobs:
           echo "🎉 Released $TAG successfully!"
           echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
 
+  # Remove the internal preview releases once the deliverable release is live.
+  # Only the Releases are deleted; the -preview.N tags stay so the validated
+  # commit remains traceable.
+  cleanup-preview-releases:
+    name: Cleanup Preview Releases
+    needs: [ build-check, publish-release ]
+    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: write
+    steps:
+      - name: Delete preview releases for this target
+        env:
+          GH_TOKEN: ${{ github.token }}
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          TAG="${{ needs.build-check.outputs.version }}"
+          RELEASES_JSON="${RUNNER_TEMP}/releases.json"
+
+          # Fetch before filtering: a failed listing must abort here instead of
+          # looking like "nothing to clean up".
+          gh api --paginate "repos/${GITHUB_REPOSITORY}/releases?per_page=100" > "$RELEASES_JSON"
+
+          # Match only <target>-preview.<digits>. String operations, not a
+          # regex over the tag, so dots in the version cannot widen the match.
+          DELETED=0
+          while IFS= read -r preview_tag; do
+            [[ -n "$preview_tag" ]] || continue
+            echo "🧹 Deleting preview release $preview_tag (tag kept)"
+            gh release delete "$preview_tag" --yes
+            DELETED=$((DELETED + 1))
+          done < <(
+            jq -r --arg tag "$TAG" '
+              .[]
+              | select(.tag_name | startswith($tag + "-preview."))
+              | select(.tag_name | ltrimstr($tag + "-preview.") | test("^[0-9]+$"))
+              | .tag_name
+            ' "$RELEASES_JSON"
+          )
+
+          if [[ "$DELETED" -eq 0 ]]; then
+            echo "ℹ️  No preview releases to clean up for $TAG"
+          else
+            echo "✅ Removed $DELETED preview release(s) for $TAG"
+          fi
+
   alert-on-failure:
     name: Alert on scheduled failure
     needs: [build-check, prepare-platform-matrix, build-rustfs, build-summary]

.github/workflows/rustfs-heal-test.yml

@@ -0,0 +1,121 @@
+name: RustFS Heal Test
+
+on:
+  workflow_dispatch:
+    inputs:
+      package_url:
+        description: 'Direct .deb URL (nightly/R2). Defaults to the latest nightly deb.'
+        required: false
+        type: string
+      stop_node_gb:
+        description: 'Stop the outage node when surviving nodes reach N GiB'
+        required: false
+        default: '15'
+      warp_stop_gb:
+        description: 'Stop warp when surviving nodes reach N GiB'
+        required: false
+        default: '40'
+      cleanup_before:
+        description: 'Reset the nodes before the test (DESTROYS existing data/config)'
+        type: boolean
+        default: true
+      cleanup_after:
+        description: 'Reset the nodes after the test (DESTROYS test data/config)'
+        type: boolean
+        default: true
+
+permissions:
+  contents: read
+
+# Only one test at a time: both this and the pool-expansion workflow mutate
+# the same test environment, so they share one concurrency group.
+concurrency:
+  group: rustfs-pool-expansion-test
+  cancel-in-progress: false
+
+defaults:
+  run:
+    shell: bash
+
+env:
+  RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
+  RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
+  RUSTFS_API_ENDPOINT: ${{ secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }}
+  RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
+  RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
+  RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
+
+jobs:
+  heal-test:
+    runs-on: smoke-testing
+    timeout-minutes: 480
+    steps:
+      - name: Checkout
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          persist-credentials: false
+
+      - name: Show environment
+        run: |
+          uname -a
+          jq --version
+          openssl version
+          warp --version || true
+          df -h /data | tail -1
+
+      - name: Reset test environment (before)
+        if: ${{ inputs.cleanup_before != 'false' }}
+        run: |
+          chmod +x scripts/test/rustfs_heal_test.sh
+          ./scripts/test/rustfs_heal_test.sh --reset -y
+
+      - name: Install RustFS package & start cluster
+        run: |
+          ARGS=(--steps "1,2" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
+          if [ -n "${{ inputs.package_url }}" ]; then
+            ARGS+=(--package-url "${{ inputs.package_url }}")
+          else
+            ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
+          fi
+          ./scripts/test/rustfs_heal_test.sh "${ARGS[@]}"
+
+      - name: Preflight checks
+        run: |
+          ARGS=(--preflight --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
+          if [ -n "${{ inputs.package_url }}" ]; then
+            ARGS+=(--package-url "${{ inputs.package_url }}")
+          else
+            ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
+          fi
+          ./scripts/test/rustfs_heal_test.sh "${ARGS[@]}"
+
+      - name: Run heal test (write -> outage -> heal -> verify)
+        run: |
+          ./scripts/test/rustfs_heal_test.sh \
+            --steps "3,4,5,6,7" -y \
+            --endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
+            --stop-node-gb "${{ inputs.stop_node_gb }}" \
+            --warp-stop-gb "${{ inputs.warp_stop_gb }}" \
+            --log-file /tmp/rustfs-heal-test.log
+
+      - name: Upload test logs
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: rustfs-heal-test-${{ github.run_id }}
+          path: |
+            /tmp/rustfs-heal-test*.log
+            /tmp/rustfs-warp.*.log
+          if-no-files-found: warn
+
+      - name: Reset test environment (after)
+        if: ${{ always() && inputs.cleanup_after != 'false' }}
+        run: |
+          ./scripts/test/rustfs_heal_test.sh --reset -y
+
+      - name: Notify on failure
+        if: failure()
+        run: |
+          echo "RustFS heal test failed"
+          echo "Package source: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
+          echo "See the uploaded log artifact for details."

.github/workflows/rustfs-performance-test.yml

@@ -0,0 +1,203 @@
+name: RustFS Performance Test
+
+on:
+  workflow_dispatch:
+    inputs:
+      package_url:
+        description: 'Direct .deb URL (nightly/R2). Defaults to the latest nightly deb.'
+        required: false
+        type: string
+      test_method:
+        description: 'Benchmark method(s) to run (manual runs only; "all" = GET+PUT+MIXED)'
+        type: choice
+        options:
+          - all
+          - get
+          - put
+          - mixed
+        default: 'all'
+      object_size:
+        description: 'Object size(s) to test (manual runs only; "all" = all 10 sizes)'
+        type: choice
+        options:
+          - all
+          - 1KiB
+          - 4KiB
+          - 16KiB
+          - 128KiB
+          - 1MiB
+          - 4MiB
+          - 8MiB
+          - 16MiB
+          - 32MiB
+          - 64MiB
+        default: 'all'
+      warp_duration:
+        description: 'warp duration per round (e.g. 5m, 30s)'
+        required: false
+        default: '5m'
+      warp_concurrency:
+        description: 'warp concurrency'
+        required: false
+        default: '64'
+      cleanup_before:
+        description: 'Reset the nodes before the test (DESTROYS existing data/config)'
+        type: boolean
+        default: true
+      cleanup_after:
+        description: 'Reset the nodes after the test (DESTROYS test data/config)'
+        type: boolean
+        default: true
+  workflow_run:
+    # Run after the nightly build completes; the nightly deb is what the test installs.
+    workflows: ["Nightly GNU Build"]
+    types: [completed]
+
+permissions:
+  contents: read
+
+# Dedicated pf-testing runner/environment: own concurrency group so perf runs
+# never block (or are blocked by) the pool-expansion / heal tests.
+concurrency:
+  group: rustfs-performance-test
+  cancel-in-progress: false
+
+defaults:
+  run:
+    shell: bash
+
+env:
+  RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
+  RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
+  # Performance test uses its own node list (4 nodes); the shared
+  # RUSTFS_NODES secret is used by the 3-node pool-expansion / heal tests.
+  RUSTFS_NODES: ${{ secrets.RUSTFS_PERF_NODES || vars.RUSTFS_PERF_NODES || 'vm000 vm001 vm002 vm003' }}
+  RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
+  # Package used by the nightly run (workflow_dispatch inputs are empty for
+  # workflow_run events), i.e. the latest nightly deb published by nightly-gnu.yml.
+  RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
+  # Fixed benchmark result directory so later steps can read summary.md
+  RUSTFS_RESULT_DIR: /tmp/rustfs-perf-results
+  # Cross-repo token for writing to rustfs/backlog (set in repo settings)
+  PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
+
+jobs:
+  performance-test:
+    runs-on: pf-testing
+    timeout-minutes: 900
+    # Run on manual dispatch, or when the nightly build completed successfully.
+    # Skipped when nightly failed.
+    if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          persist-credentials: false
+          ref: ${{ github.event.workflow_run.head_sha || github.ref }}
+
+      - name: Show environment
+        run: |
+          uname -a
+          jq --version
+          warp --version || true
+          df -h /data | tail -1
+
+      - name: Reset test environment (before)
+        if: ${{ inputs.cleanup_before != 'false' }}
+        run: |
+          chmod +x scripts/test/rustfs_performance_test.sh
+          ./scripts/test/rustfs_performance_test.sh --step 1 -y
+
+      - name: Install RustFS package & start cluster (4x4)
+        run: |
+          ARGS=(--steps "2,3,4" -y)
+          if [ -n "${{ inputs.package_url }}" ]; then
+            ARGS+=(--package-url "${{ inputs.package_url }}")
+          else
+            ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
+          fi
+          ./scripts/test/rustfs_performance_test.sh "${ARGS[@]}"
+
+      - name: Preflight checks
+        run: |
+          ARGS=(--preflight)
+          if [ -n "${{ inputs.package_url }}" ]; then
+            ARGS+=(--package-url "${{ inputs.package_url }}")
+          else
+            ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
+          fi
+          ./scripts/test/rustfs_performance_test.sh "${ARGS[@]}"
+
+      - name: Run benchmark (GET/PUT/MIXED)
+        id: benchmark
+        run: |
+          # Empty on automatic (workflow_run) runs -> full 30 rounds.
+          # Manual dispatch can restrict method(s)/size(s).
+          export WARP_METHODS="${{ inputs.test_method }}"
+          export WARP_SIZES="${{ inputs.object_size }}"
+          ./scripts/test/rustfs_performance_test.sh \
+            --step 5 -y \
+            --warp-duration "${{ inputs.warp_duration || '5m' }}" \
+            --warp-concurrency "${{ inputs.warp_concurrency || '64' }}" \
+            --log-file /tmp/rustfs-perf-test.log
+
+      - name: Analyze results
+        if: ${{ steps.benchmark.conclusion == 'success' }}
+        run: |
+          ./scripts/test/rustfs_performance_test.sh --step 6 -y
+
+      - name: Post results to backlog issue
+        if: ${{ steps.benchmark.conclusion == 'success' }}
+        env:
+          GH_TOKEN: ${{ env.PF_TESTING_GH_TOKEN }}
+          RESULT_DIR: ${{ env.RUSTFS_RESULT_DIR }}
+        run: |
+          set -euo pipefail
+          if [ -z "${GH_TOKEN:-}" ]; then
+            echo "PF_TESTING_GH_TOKEN is not configured; skipping issue post"
+            exit 0
+          fi
+          SUMMARY="${RESULT_DIR}/summary.md"
+          [ -f "${SUMMARY}" ] || { echo "summary.md not found at ${SUMMARY}"; exit 1; }
+          DATE="$(date -u +%Y-%m-%d)"
+          {
+            echo "## RustFS nightly build performance testing report"
+            echo ""
+            echo "- **日期**: ${DATE}"
+            echo "- **Run**: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+            echo "- **触发方式**: ${{ github.event_name }}"
+            echo ""
+            cat "${SUMMARY}"
+          } > /tmp/rustfs-perf-issue-body.md
+          TITLE="RustFS nightly build performance testing report"
+          EXISTING="$(gh issue list --repo rustfs/backlog \
+            --search "in:title \"${TITLE}\"" --state all --limit 5 \
+            --json number --jq '.[0].number // empty')"
+          if [ -n "${EXISTING}" ]; then
+            gh issue comment "${EXISTING}" --repo rustfs/backlog --body-file /tmp/rustfs-perf-issue-body.md
+            echo "commented on existing issue #${EXISTING}"
+          else
+            gh issue create --repo rustfs/backlog --title "${TITLE}" --body-file /tmp/rustfs-perf-issue-body.md
+          fi
+
+      - name: Upload test logs & results
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: rustfs-perf-test-${{ github.run_id }}
+          path: |
+            /tmp/rustfs-perf-test*.log
+            /tmp/rustfs-perf-results/**
+          if-no-files-found: warn
+
+      - name: Reset test environment (after)
+        if: ${{ always() && inputs.cleanup_after != 'false' }}
+        run: |
+          ./scripts/test/rustfs_performance_test.sh --step 7 -y
+
+      - name: Notify on failure
+        if: failure()
+        run: |
+          echo "RustFS performance test failed"
+          echo "Package source: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
+          echo "See the uploaded log artifact for details."

.github/workflows/rustfs-pool-expand-test.yml

@@ -30,6 +30,14 @@ on:
         description: 'Run the pool decommission step (3-pool topology only)'
         type: boolean
         default: true
+      stop_node_gb:
+        description: 'Heal: stop the outage node when surviving nodes reach N GiB'
+        required: false
+        default: '15'
+      warp_stop_gb:
+        description: 'Heal: stop warp when surviving nodes reach N GiB'
+        required: false
+        default: '40'
       cleanup_before:
         description: 'Reset the nodes before the test (DESTROYS existing data/config)'
         type: boolean
@@ -38,9 +46,10 @@ on:
         description: 'Reset the nodes after the test (DESTROYS test data/config)'
         type: boolean
         default: true
-  schedule:
-    # Nightly regression run; remove if you do not want a schedule.
-    - cron: '0 21 * * *'
+  workflow_run:
+    # Run after the nightly build completes: pool expansion first, then heal.
+    workflows: ["Nightly GNU Build"]
+    types: [completed]
 
 permissions:
   contents: read
@@ -61,19 +70,23 @@ env:
   RUSTFS_API_ENDPOINT: ${{ secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }}
   RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
   RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
-  # Package used by the scheduled run (workflow_dispatch inputs are empty for
-  # schedule events), i.e. the latest nightly deb published by nightly-gnu.yml.
+  # Package used by the nightly run (workflow_dispatch inputs are empty for
+  # workflow_run events), i.e. the latest nightly deb published by nightly-gnu.yml.
   RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
 
 jobs:
   pool-expansion-test:
     runs-on: smoke-testing
     timeout-minutes: 360
+    # Run on manual dispatch, or when the nightly build completed successfully
+    # (its deb is what the tests install). Skipped when nightly failed.
+    if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
     steps:
       - name: Checkout
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
         with:
           persist-credentials: false
+          ref: ${{ github.event.workflow_run.head_sha || github.ref }}
 
       - name: Show environment
         run: |
@@ -91,7 +104,7 @@ jobs:
 
       - name: Install RustFS package & start first pool
         run: |
-          ARGS=(--steps 1,2,3 -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
+          ARGS=(--steps "1,2,3" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
           if [ -n "${{ inputs.package_url }}" ]; then
             ARGS+=(--package-url "${{ inputs.package_url }}")
           elif [ -n "${{ inputs.rustfs_version }}" ]; then
@@ -137,8 +150,8 @@ jobs:
         with:
           name: rustfs-pool-test-${{ github.run_id }}
           path: |
-            /tmp/rustfs-pool-test.log
-            /tmp/rustfs-warp.log
+            /tmp/rustfs-pool-test*.log
+            /tmp/rustfs-warp.*.log
           if-no-files-found: warn
 
       - name: Reset test environment (after)
@@ -152,3 +165,74 @@ jobs:
           echo "RustFS pool expansion test failed"
           echo "Package source: ${{ inputs.package_url || inputs.rustfs_version || 'nightly (R2 latest)' }}"
           echo "See the uploaded log artifact for details."
+
+  # Heal regression runs after the pool test regardless of its outcome: a pool
+  # failure must be reported (it makes the run red) but must not block heal.
+  heal-test:
+    name: Heal test (after pool test)
+    runs-on: smoke-testing
+    timeout-minutes: 480
+    needs: pool-expansion-test
+    if: ${{ always() && (github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success') }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          persist-credentials: false
+          ref: ${{ github.event.workflow_run.head_sha || github.ref }}
+
+      - name: Reset test environment (before)
+        if: ${{ inputs.cleanup_before != 'false' }}
+        run: |
+          chmod +x scripts/test/rustfs_heal_test.sh
+          ./scripts/test/rustfs_heal_test.sh --reset -y
+
+      - name: Install RustFS package & start cluster
+        run: |
+          ARGS=(--steps "1,2" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
+          if [ -n "${{ inputs.package_url }}" ]; then
+            ARGS+=(--package-url "${{ inputs.package_url }}")
+          else
+            ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
+          fi
+          ./scripts/test/rustfs_heal_test.sh "${ARGS[@]}"
+
+      - name: Preflight checks
+        run: |
+          ARGS=(--preflight --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
+          if [ -n "${{ inputs.package_url }}" ]; then
+            ARGS+=(--package-url "${{ inputs.package_url }}")
+          else
+            ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
+          fi
+          ./scripts/test/rustfs_heal_test.sh "${ARGS[@]}"
+
+      - name: Run heal test (write -> outage -> heal -> verify)
+        run: |
+          ./scripts/test/rustfs_heal_test.sh \
+            --steps 3,4,5,6,7 -y \
+            --endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
+            --stop-node-gb "${{ inputs.stop_node_gb || '15' }}" \
+            --warp-stop-gb "${{ inputs.warp_stop_gb || '40' }}" \
+            --log-file /tmp/rustfs-heal-test.log
+
+      - name: Upload test logs
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: rustfs-heal-test-${{ github.run_id }}
+          path: |
+            /tmp/rustfs-heal-test.log
+            /tmp/rustfs-warp.*.log
+          if-no-files-found: warn
+
+      - name: Reset test environment (after)
+        if: ${{ always() && inputs.cleanup_after != 'false' }}
+        run: |
+          ./scripts/test/rustfs_heal_test.sh --reset -y
+
+      - name: Notify on failure
+        if: failure()
+        run: |
+          echo "RustFS heal test failed"
+          echo "See the uploaded log artifact for details."

Cargo.lock

@@ -91,7 +91,6 @@ dependencies = [
  "const-random",
  "getrandom 0.3.4",
  "once_cell",
- "serde",
  "version_check",
  "zerocopy",
 ]
@@ -9627,7 +9626,6 @@ dependencies = [
 name = "rustfs-ecstore"
 version = "1.0.0-rc.4"
 dependencies = [
- "ahash",
  "arc-swap",
  "async-channel",
  "async-recursion",
@@ -9773,7 +9771,6 @@ dependencies = [
 name = "rustfs-filemeta"
 version = "1.0.0-rc.4"
 dependencies = [
- "ahash",
  "arc-swap",
  "byteorder",
  "bytes",
@@ -10814,7 +10811,6 @@ dependencies = [
 name = "rustfs-utils"
 version = "1.0.0-rc.4"
 dependencies = [
- "ahash",
  "base64-simd",
  "blake2",
  "brotli",

Cargo.toml

@@ -368,9 +368,6 @@ hotpath = { version = "0.24.0", default-features = false }
 # Snapshot testing for output format regression detection
 insta = { version = "1.48" }
 
-# High-performance hashing
-ahash = { version = "0.8", default-features = false, features = ["std", "runtime-rng", "serde"] }
-
 [workspace.metadata.cargo-shear]
 ignored = ["hotpath", "rustfs"]
 

crates/config/src/constants/health.rs

@@ -40,6 +40,14 @@ pub const HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS: u64 = 5_000;
 pub const ENV_HEALTH_CLUSTER_TIMEOUT_MS: &str = "RUSTFS_HEALTH_CLUSTER_TIMEOUT_MS";
 pub const DEFAULT_HEALTH_CLUSTER_TIMEOUT_MS: u64 = 2000;
 
+/// Timeout for one remote lock-client online check used by readiness (milliseconds).
+///
+/// This is intentionally shorter than the generic lock RPC timeout so
+/// `/health/ready` can report degradation instead of riding a dead peer's
+/// connect or HTTP/2 keepalive budget.
+pub const ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS: &str = "RUSTFS_HEALTH_LOCK_ONLINE_TIMEOUT_MS";
+pub const DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS: u64 = 1000;
+
 /// Maximum time to wait for local node runtime readiness (storage / IAM / lock
 /// quorum) during startup before failing fast (seconds).
 ///

crates/config/src/constants/object.rs

@@ -288,6 +288,39 @@ pub const DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 0;
 
 const _: () = assert!(!DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE);
 
+/// Enable automatic foreground admission for large or unknown-size PutObject requests.
+///
+/// Unlike the strict experimental gate above, this default-on path only applies
+/// to requests that are large enough to create sustained erasure/RPC pressure.
+/// Small PUTs continue on the legacy path unless the strict gate is explicitly
+/// enabled.
+pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE";
+pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: bool = true;
+
+/// Maximum large foreground PutObject requests admitted concurrently per process.
+///
+/// `0` derives a conservative default from the local disk-read scheduler cap,
+/// currently clamped to protect the commit path without making ordinary high
+/// throughput uploads single-file.
+pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT";
+pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: usize = 0;
+
+/// Minimum object size that enters automatic large PutObject admission.
+///
+/// Requests with an unknown size are treated as large because the write pressure
+/// cannot be bounded from headers.
+pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES";
+pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 32 * 1024 * 1024;
+
+/// Time in milliseconds a large foreground PutObject waits for a permit.
+///
+/// A short wait smooths transient bursts while still returning S3
+/// `SlowDown`/503 before body ingest when the node is already saturated.
+pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
+pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 250;
+
+const _: () = assert!(DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE);
+
 /// Environment variable for minimum GetObject timeout in seconds.
 ///
 /// When dynamic timeout calculation is enabled, this is the minimum timeout

crates/e2e_test/src/kms/kms_anonymous_enforcement_test.rs

@@ -0,0 +1,220 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Anonymous access to SSE-KMS objects under per-key authorization.
+//!
+//! Locks both halves of the anonymous contract decided in backlog#2028 (D4):
+//!
+//! - **Enforcement on**: anonymous requests hold no `kms` grants, so a public
+//!   bucket policy does not let them read SSE-KMS objects or write through an
+//!   SSE-KMS default-encryption rule. Both fail with `AccessDenied`.
+//! - **Enforcement off** (the default): bucket policy alone governs anonymous
+//!   access, matching the pre-enforcement behavior — public SSE-KMS objects are
+//!   decrypted and served, and anonymous writes are encrypted under the default
+//!   key.
+//!
+//! The denial today is emergent — an empty-account principal falling through to
+//! the IAM default deny — so without this file a refactor of principal
+//! construction or policy evaluation could silently flip it. Each test carries a
+//! plaintext-object positive control: a denial proves nothing while the bucket
+//! policy has not propagated.
+
+use super::common::{LocalKMSTestEnvironment, create_key_with_specific_id};
+use crate::common::{init_logging, local_http_client};
+use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::types::{
+    ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
+};
+use std::time::Duration;
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+const DEFAULT_KEY: &str = "kms-anon-default-key";
+const BUCKET: &str = "kms-anon-enforcement";
+const PLAIN_OBJECT: &str = "plain.txt";
+const ENCRYPTED_OBJECT: &str = "encrypted.txt";
+const PAYLOAD: &[u8] = b"kms anonymous enforcement payload";
+
+/// How long a bucket policy change may take to reach the request path.
+const POLICY_PROPAGATION: Duration = Duration::from_secs(20);
+
+/// Start a local-KMS server and build the public-bucket fixture.
+///
+/// The bucket holds a plaintext object (the positive control), an SSE-KMS
+/// object, an SSE-KMS default-encryption rule, and a bucket policy opening
+/// `GetObject`/`PutObject` to everyone. The enforcement switch defaults to off,
+/// so the enforcing case has to set it explicitly.
+async fn start_public_sse_kms_bucket(env: &mut LocalKMSTestEnvironment, enforce: bool) -> TestResult {
+    create_key_with_specific_id(&env.kms_keys_dir, DEFAULT_KEY).await?;
+
+    let key_dir = env.kms_keys_dir.clone();
+    let args = vec![
+        "--kms-enable",
+        "--kms-backend",
+        "local",
+        "--kms-key-dir",
+        key_dir.as_str(),
+        "--kms-default-key-id",
+        DEFAULT_KEY,
+    ];
+    let mut envs = vec![("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")];
+    if enforce {
+        envs.push(("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "true"));
+    }
+    env.base_env.start_rustfs_server_with_env(args, &envs).await?;
+    env.base_env.create_test_bucket(BUCKET).await?;
+
+    let owner = env.base_env.create_s3_client();
+
+    owner
+        .put_object()
+        .bucket(BUCKET)
+        .key(PLAIN_OBJECT)
+        .body(ByteStream::from_static(PAYLOAD))
+        .send()
+        .await?;
+    owner
+        .put_object()
+        .bucket(BUCKET)
+        .key(ENCRYPTED_OBJECT)
+        .body(ByteStream::from_static(PAYLOAD))
+        .server_side_encryption(ServerSideEncryption::AwsKms)
+        .ssekms_key_id(DEFAULT_KEY)
+        .send()
+        .await?;
+
+    let encryption_config = ServerSideEncryptionConfiguration::builder()
+        .rules(
+            ServerSideEncryptionRule::builder()
+                .apply_server_side_encryption_by_default(
+                    ServerSideEncryptionByDefault::builder()
+                        .sse_algorithm(ServerSideEncryption::AwsKms)
+                        .kms_master_key_id(DEFAULT_KEY)
+                        .build()?,
+                )
+                .build(),
+        )
+        .build()?;
+    owner
+        .put_bucket_encryption()
+        .bucket(BUCKET)
+        .server_side_encryption_configuration(encryption_config)
+        .send()
+        .await?;
+
+    let policy = serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [{
+            "Sid": "PublicReadWrite",
+            "Effect": "Allow",
+            "Principal": "*",
+            "Action": ["s3:GetObject", "s3:PutObject"],
+            "Resource": [format!("arn:aws:s3:::{BUCKET}/*")]
+        }]
+    })
+    .to_string();
+    owner.put_bucket_policy().bucket(BUCKET).policy(&policy).send().await?;
+    let _ = owner.delete_public_access_block().bucket(BUCKET).send().await;
+
+    Ok(())
+}
+
+fn object_url(env: &LocalKMSTestEnvironment, key: &str) -> String {
+    format!("{}/{BUCKET}/{key}", env.base_env.url)
+}
+
+async fn anonymous_get(env: &LocalKMSTestEnvironment, key: &str) -> Result<reqwest::Response, reqwest::Error> {
+    local_http_client().get(object_url(env, key)).send().await
+}
+
+async fn anonymous_put(env: &LocalKMSTestEnvironment, key: &str) -> Result<reqwest::Response, reqwest::Error> {
+    local_http_client().put(object_url(env, key)).body(PAYLOAD).send().await
+}
+
+/// Retry the plaintext read until the public bucket policy is live.
+async fn wait_for_public_read(env: &LocalKMSTestEnvironment) -> TestResult {
+    let deadline = tokio::time::Instant::now() + POLICY_PROPAGATION;
+    loop {
+        let status = anonymous_get(env, PLAIN_OBJECT).await?.status();
+        if status.as_u16() == 200 {
+            return Ok(());
+        }
+        if tokio::time::Instant::now() >= deadline {
+            return Err(format!("positive control never became readable: anonymous GET {PLAIN_OBJECT} -> {status}").into());
+        }
+        tokio::time::sleep(Duration::from_millis(500)).await;
+    }
+}
+
+async fn assert_anonymous_denied(response: reqwest::Response, what: &str) -> TestResult {
+    let status = response.status().as_u16();
+    let body = response.text().await?;
+    assert_eq!(status, 403, "{what} must be denied, got {status}: {body}");
+    assert!(body.contains("AccessDenied"), "{what} must carry AccessDenied: {body}");
+    Ok(())
+}
+
+/// Enforcement on: a public bucket policy does not exempt anonymous requests
+/// from per-key authorization, on either the read or the default-encryption
+/// write path.
+#[tokio::test(flavor = "multi_thread")]
+async fn anonymous_sse_kms_denied_under_enforcement() -> TestResult {
+    init_logging();
+
+    let mut env = LocalKMSTestEnvironment::new().await?;
+    start_public_sse_kms_bucket(&mut env, true).await?;
+    wait_for_public_read(&env).await?;
+
+    let read = anonymous_get(&env, ENCRYPTED_OBJECT).await?;
+    assert_anonymous_denied(read, "anonymous GET of an SSE-KMS object").await?;
+
+    let write = anonymous_put(&env, "anon-write.txt").await?;
+    assert_anonymous_denied(write, "anonymous PUT through an SSE-KMS default-encryption rule").await?;
+
+    Ok(())
+}
+
+/// Enforcement off (the default): bucket policy alone governs anonymous access,
+/// and the default-encryption rule still encrypts anonymous writes.
+#[tokio::test(flavor = "multi_thread")]
+async fn anonymous_sse_kms_governed_by_bucket_policy_without_enforcement() -> TestResult {
+    init_logging();
+
+    let mut env = LocalKMSTestEnvironment::new().await?;
+    start_public_sse_kms_bucket(&mut env, false).await?;
+    wait_for_public_read(&env).await?;
+
+    let read = anonymous_get(&env, ENCRYPTED_OBJECT).await?;
+    assert_eq!(read.status().as_u16(), 200, "anonymous GET of a public SSE-KMS object must succeed");
+    assert_eq!(read.bytes().await?.as_ref(), PAYLOAD, "the object must be served decrypted");
+
+    let write = anonymous_put(&env, "anon-write.txt").await?;
+    assert_eq!(write.status().as_u16(), 200, "anonymous PUT to a public bucket must succeed");
+
+    let stored = env
+        .base_env
+        .create_s3_client()
+        .head_object()
+        .bucket(BUCKET)
+        .key("anon-write.txt")
+        .send()
+        .await?;
+    assert_eq!(
+        stored.server_side_encryption(),
+        Some(&ServerSideEncryption::AwsKms),
+        "the anonymous write must be encrypted by the bucket default rule"
+    );
+
+    Ok(())
+}

crates/e2e_test/src/kms/mod.rs

@@ -57,6 +57,9 @@ mod copy_object_version_restore_sse_test;
 #[cfg(test)]
 mod configured_roundtrip_test;
 
+#[cfg(test)]
+mod kms_anonymous_enforcement_test;
+
 #[cfg(test)]
 mod kms_authorization_negative_matrix_test;
 

crates/ecstore/Cargo.toml

@@ -218,9 +218,6 @@ faster-hex = { workspace = true }
 ratelimit = { workspace = true }
 aws-smithy-http-client = { workspace = true, default-features = false, features = ["rustls-aws-lc"] }
 
-# High-performance hashing
-ahash = { workspace = true, features = ["serde"] }
-
 # Observability and Metrics
 metrics = { workspace = true }
 

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -12,7 +12,6 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use ahash::AHashMap;
 use super::{metadata_boundary, object_lock_boundary, runtime_boundary as runtime_sources};
 use crate::bucket::lifecycle::bucket_lifecycle_audit::{
     LcAuditEvent, LcEventSrc, emit_non_transitioned_expiration_event, emit_transition_complete_event,
@@ -2871,7 +2870,7 @@ fn spawn_transition_transaction_recovery_once(api: Arc<ECStore>) {
 struct StaleMultipartUploadCandidate {
     path: String,
     initiated: OffsetDateTime,
-    metadata: Option<AHashMap<String, String>>,
+    metadata: Option<HashMap<String, String>>,
 }
 
 fn parse_stale_uploads_duration(env_key: &str, default: StdDuration) -> StdDuration {
@@ -2916,9 +2915,9 @@ async fn stale_upload_current_size(set: &Arc<SetDisks>, metadata: &HashMap<Strin
     stale_upload_current_size_with_opts(set, metadata, upload_dir, false).await
 }
 
-async fn stale_upload_current_size_with_opts<S: std::hash::BuildHasher>(
+async fn stale_upload_current_size_with_opts(
     set: &Arc<SetDisks>,
-    metadata: &HashMap<String, String, S>,
+    metadata: &HashMap<String, String>,
     upload_dir: &str,
     no_lock: bool,
 ) -> Option<usize> {
@@ -2951,9 +2950,9 @@ async fn stale_upload_current_size_with_opts<S: std::hash::BuildHasher>(
     )
 }
 
-async fn stale_upload_lifecycle_due<S: std::hash::BuildHasher>(
+async fn stale_upload_lifecycle_due(
     set: &Arc<SetDisks>,
-    metadata: &HashMap<String, String, S>,
+    metadata: &HashMap<String, String>,
     initiated: OffsetDateTime,
     upload_dir: &str,
     no_lock: bool,
@@ -2979,7 +2978,7 @@ async fn stale_upload_lifecycle_due<S: std::hash::BuildHasher>(
             .unwrap_or_default(),
         is_latest: true,
         delete_marker: false,
-        user_defined: metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect(),
+        user_defined: metadata.clone(),
         ..Default::default()
     };
 

crates/ecstore/src/cluster/rpc/remote_locker.rs

@@ -27,7 +27,7 @@ use rustfs_protos::{
     ConnectionEvictionLogLevel, evict_failed_connection_with_log_level, models::PingBodyBuilder,
     proto_gen::node_service::node_service_client::NodeServiceClient,
 };
-use std::time::Duration;
+use std::{sync::OnceLock, time::Duration};
 use tokio::time::timeout;
 use tonic::Request;
 use tonic::service::interceptor::InterceptedService;
@@ -44,11 +44,35 @@ pub struct RemoteClient {
 }
 
 impl RemoteClient {
+    const ONLINE_CHECK_RESOURCE: &'static str = "health-lock-online";
+
     pub fn new(endpoint: String) -> Self {
         Self { addr: endpoint }
     }
 
+    fn ping_body() -> Bytes {
+        static BODY: OnceLock<Bytes> = OnceLock::new();
+        BODY.get_or_init(|| {
+            let mut fbb = flatbuffers::FlatBufferBuilder::new();
+            let payload = fbb.create_vector(b"health-check");
+            let mut builder = PingBodyBuilder::new(&mut fbb);
+            builder.add_payload(payload);
+            let root = builder.finish();
+            fbb.finish(root, None);
+            Bytes::copy_from_slice(fbb.finished_data())
+        })
+        .clone()
+    }
+
     fn build_ping_request() -> PingRequest {
+        PingRequest {
+            version: 1,
+            body: Self::ping_body(),
+        }
+    }
+
+    #[cfg(test)]
+    fn build_fresh_ping_request_for_test() -> PingRequest {
         let mut fbb = flatbuffers::FlatBufferBuilder::new();
         let payload = fbb.create_vector(b"health-check");
         let mut builder = PingBodyBuilder::new(&mut fbb);
@@ -164,6 +188,16 @@ impl RemoteClient {
         )
     }
 
+    fn online_check_timeout() -> Duration {
+        Duration::from_millis(
+            rustfs_utils::get_env_u64(
+                rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS,
+                rustfs_config::DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS,
+            )
+            .max(1),
+        )
+    }
+
     async fn execute_rpc<T, F>(&self, op: &'static str, resource_summary: &str, future: F) -> std::result::Result<T, LockError>
     where
         F: std::future::Future<Output = std::result::Result<T, tonic::Status>>,
@@ -547,24 +581,37 @@ impl LockClient for RemoteClient {
     }
 
     async fn is_online(&self) -> bool {
-        // Use Ping interface to test if remote service is online
-        let mut client = match self.get_client().await {
-            Ok(client) => client,
-            Err(_) => {
-                info!("remote client {} connection failed", self.addr);
-                return false;
-            }
-        };
-
-        let ping_req = Request::new(Self::build_ping_request());
-
-        match client.ping(ping_req).await {
-            Ok(_) => {
-                info!("remote client {} is online", self.addr);
+        let online_timeout = Self::online_check_timeout();
+        match timeout(online_timeout, async {
+            let mut client = self.get_client().await?;
+            let ping_req = Request::new(Self::build_ping_request());
+            self.execute_rpc("ping", Self::ONLINE_CHECK_RESOURCE, client.ping(ping_req))
+                .await?;
+            Ok::<(), LockError>(())
+        })
+        .await
+        {
+            Ok(Ok(())) => {
+                debug!(addr = %self.addr, timeout_ms = online_timeout.as_millis(), "remote lock client is online");
                 true
             }
+            Ok(Err(err)) => {
+                debug!(
+                    addr = %self.addr,
+                    timeout_ms = online_timeout.as_millis(),
+                    error = %err,
+                    "remote lock client online check failed"
+                );
+                false
+            }
             Err(_) => {
-                info!("remote client {} ping failed", self.addr);
+                let reason = format!("online check timed out after {:?}", online_timeout);
+                warn!(
+                    addr = %self.addr,
+                    timeout_ms = online_timeout.as_millis(),
+                    "remote lock client online check timed out"
+                );
+                self.evict_connection("ping", &reason, Self::ONLINE_CHECK_RESOURCE).await;
                 false
             }
         }
@@ -651,6 +698,15 @@ mod tests {
         );
     }
 
+    #[test]
+    fn cached_ping_request_matches_fresh_flatbuffer_payload() {
+        let cached = RemoteClient::build_ping_request();
+        let fresh = RemoteClient::build_fresh_ping_request_for_test();
+
+        assert_eq!(cached.version, fresh.version);
+        assert_eq!(cached.body, fresh.body);
+    }
+
     #[tokio::test]
     #[serial_test::serial]
     async fn test_remote_client_acquire_lock_uses_rpc_timeout_and_evicts_connection() {
@@ -779,6 +835,48 @@ mod tests {
         accept_task.abort();
     }
 
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn test_remote_client_is_online_uses_health_timeout_and_evicts_connection() {
+        ensure_test_rpc_secret();
+        let Some((addr, accept_task)) = spawn_hanging_listener().await else {
+            return;
+        };
+        cache_lazy_channel(&addr).await;
+        assert!(runtime_sources::test_node_channel_is_cached(&addr).await);
+
+        temp_env::async_with_vars(
+            [
+                (rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("50")),
+                (rustfs_config::ENV_OBJECT_LOCK_RPC_TIMEOUT_MS, Some("1000")),
+            ],
+            async {
+                let client = RemoteClient::new(addr.clone());
+                let started_at = tokio::time::Instant::now();
+
+                let online = client.is_online().await;
+                let elapsed = started_at.elapsed();
+
+                assert!(!online, "hanging remote lock peer must not be reported online");
+                assert!(
+                    elapsed >= Duration::from_millis(40),
+                    "remote online check should honor configured health timeout, got {elapsed:?}"
+                );
+                assert!(
+                    elapsed < Duration::from_secs(1),
+                    "health timeout should keep readiness probes bounded, got {elapsed:?}"
+                );
+                assert!(
+                    !runtime_sources::test_node_channel_is_cached(&addr).await,
+                    "online-check timeout should evict cached connection"
+                );
+            },
+        )
+        .await;
+
+        accept_task.abort();
+    }
+
     #[tokio::test]
     #[serial_test::serial]
     async fn test_remote_client_refresh_tonic_error_evicts_connection() {
@@ -906,4 +1004,21 @@ mod tests {
             assert_eq!(RemoteClient::rpc_timeout(), Duration::from_millis(1));
         });
     }
+
+    #[test]
+    #[serial_test::serial]
+    fn test_remote_client_online_timeout_honors_configured_deadline() {
+        temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, None::<&str>, || {
+            assert_eq!(
+                RemoteClient::online_check_timeout(),
+                Duration::from_millis(rustfs_config::DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS)
+            );
+        });
+        temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("50"), || {
+            assert_eq!(RemoteClient::online_check_timeout(), Duration::from_millis(50));
+        });
+        temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("0"), || {
+            assert_eq!(RemoteClient::online_check_timeout(), Duration::from_millis(1));
+        });
+    }
 }

crates/ecstore/src/config/com.rs

@@ -2793,12 +2793,13 @@ where
 #[cfg(test)]
 mod tests {
     use super::{
-        SERVER_CONFIG_LOCK, ServerConfigSnapshot, apply_dynamic_config_for_sub_sys_with, config_task_join_error,
-        configs_semantically_equal, decode_server_config_blob, encode_server_config_blob, is_standard_object_server_config,
-        lookup_configs, new_and_save_server_config, read_config, read_config_no_lock_preserve_empty_with_metadata,
-        read_config_preserve_empty, read_config_with_metadata, read_config_without_migrate, read_server_config_snapshot,
-        save_server_config, save_server_config_snapshot, save_server_config_snapshot_with_generation,
-        server_config_transaction_lock_path, should_warn_ignored_scalar_section, storage_class_kvs_mut,
+        SERVER_CONFIG_LOCK, ServerConfigSnapshot, apply_dynamic_config_for_sub_sys_with, build_scalar_config_object,
+        config_task_join_error, configs_semantically_equal, decode_server_config_blob, encode_server_config_blob,
+        heal_config_descriptor, is_standard_object_server_config, lookup_configs, new_and_save_server_config, read_config,
+        read_config_no_lock_preserve_empty_with_metadata, read_config_preserve_empty, read_config_with_metadata,
+        read_config_without_migrate, read_server_config_snapshot, save_server_config, save_server_config_snapshot,
+        save_server_config_snapshot_with_generation, server_config_transaction_lock_path, should_warn_ignored_scalar_section,
+        storage_class_kvs_mut,
     };
     use crate::config::{audit, heal, notify, oidc, scanner};
     use crate::disk::endpoint::Endpoint;
@@ -3541,6 +3542,31 @@ mod tests {
         cfg
     }
 
+    /// `Config::new()` (and every decode built on it) reads the process-global
+    /// `rustfs_config::server_config::DEFAULT_KVS` OnceLock at call time, and
+    /// other tests in this binary register it via `crate::config::init()`
+    /// mid-run. Equality assertions must therefore normalize both sides with
+    /// one snapshot taken after both configs exist, never against a later
+    /// `Config::new()`.
+    fn default_kvs_snapshot() -> Option<&'static std::collections::HashMap<String, KVS>> {
+        rustfs_config::server_config::DEFAULT_KVS.get()
+    }
+
+    /// Fills the default sections `cfg` is missing from an explicit
+    /// [`default_kvs_snapshot`], mirroring `Config::set_defaults`.
+    fn filled_with_default_kvs(mut cfg: Config, snapshot: Option<&std::collections::HashMap<String, KVS>>) -> Config {
+        if let Some(defaults) = snapshot {
+            for (sub_sys, kvs) in defaults {
+                cfg.0
+                    .entry(sub_sys.clone())
+                    .or_default()
+                    .entry(DEFAULT_DELIMITER.to_string())
+                    .or_insert_with(|| kvs.clone());
+            }
+        }
+        cfg
+    }
+
     #[test]
     fn test_external_scanner_config_decodes_with_defaults() {
         let cfg =
@@ -3630,7 +3656,9 @@ mod tests {
         }"#;
 
         let cfg = decode_server_config_blob(seed).expect("root heal null should mean no persisted override");
-        assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
+        // The heal section may hold registered defaults, so assert on the
+        // semantic diff instead of the section's presence.
+        assert!(build_scalar_config_object(&cfg, heal_config_descriptor()).is_empty());
         assert!(!is_standard_object_server_config(seed));
 
         let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("legacy seed should canonicalize on an authorized save");
@@ -3665,7 +3693,12 @@ mod tests {
             let input = format!(r#"{{"version":"33","storageclass":{{"standard":"","rrs":""}},{section}}}"#);
             let cfg = decode_server_config_blob(input.as_bytes())
                 .unwrap_or_else(|err| panic!("legacy scalar section {section} should be ignored, got: {err}"));
-            assert_eq!(cfg, base, "ignored section {section} must contribute no overrides");
+            let snapshot = default_kvs_snapshot();
+            assert_eq!(
+                filled_with_default_kvs(cfg.clone(), snapshot),
+                filled_with_default_kvs(base.clone(), snapshot),
+                "ignored section {section} must contribute no overrides"
+            );
             assert!(
                 !is_standard_object_server_config(input.as_bytes()),
                 "seed with {section} must not count as standard so a save rewrites it"
@@ -3743,12 +3776,19 @@ mod tests {
     fn valid_heal_object_and_kvs_array_shapes_remain_accepted() {
         let empty_object = br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":{}}"#;
         let cfg = decode_server_config_blob(empty_object).expect("empty heal object should decode as no override");
-        assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
+        // The heal section may hold registered defaults, so assert on the
+        // semantic diff instead of the section's presence.
+        assert!(build_scalar_config_object(&cfg, heal_config_descriptor()).is_empty());
 
         let kvs_array =
             br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":[{"key":"bitrot_cycle","value":"off"}]}"#;
         let cfg = decode_server_config_blob(kvs_array).expect("heal KVS array should decode");
-        assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_some());
+        assert_eq!(
+            build_scalar_config_object(&cfg, heal_config_descriptor())
+                .get(HEAL_BITROT_CYCLE)
+                .and_then(Value::as_str),
+            Some("off")
+        );
     }
 
     #[test]
@@ -4900,8 +4940,12 @@ mod tests {
     fn test_fallback_returns_default_config_when_recovery_enabled() {
         let cfg = fallback_server_config_after_corruption(corrupt_config_error(), "config/config.json", true)
             .expect("recovery enabled must fall back to the default config");
+        let snapshot = default_kvs_snapshot();
         assert!(
-            configs_semantically_equal(&cfg, &Config::new()),
+            configs_semantically_equal(
+                &filled_with_default_kvs(cfg, snapshot),
+                &filled_with_default_kvs(Config(std::collections::HashMap::new()), snapshot)
+            ),
             "fallback config should be the default server config"
         );
     }
@@ -5518,8 +5562,12 @@ mod tests {
             .expect("unrecoverable corruption should fall back to the default config");
 
         assert_eq!(store.heal_calls.load(Ordering::SeqCst), 1, "heal should be attempted before falling back");
+        let snapshot = default_kvs_snapshot();
         assert!(
-            configs_semantically_equal(&cfg, &Config::new()),
+            configs_semantically_equal(
+                &filled_with_default_kvs(cfg, snapshot),
+                &filled_with_default_kvs(Config(std::collections::HashMap::new()), snapshot)
+            ),
             "fallback config should be the default server config"
         );
     }

crates/ecstore/src/core/pools.rs

@@ -96,6 +96,8 @@ const LOG_SUBSYSTEM_POOLS: &str = "pools";
 const EVENT_DECOMMISSION_STATE: &str = "decommission_state";
 const EVENT_DECOMMISSION_BUCKET: &str = "decommission_bucket";
 const EVENT_DECOMMISSION_ENTRY: &str = "decommission_entry";
+const POOL_ACTIVATION_FLEET_PROOF_REQUIRED: &str = "pool activation requires a live fleet capability proof";
+const POOL_ACTIVATION_FLEET_PROOF_EXPIRED: &str = "pool activation fleet capability proof expired before commit";
 const DECOMMISSION_STAGE_MIGRATE_OBJECT: &str = "migrate_object";
 const DECOMMISSION_STAGE_CLEANUP_PREFLIGHT: &str = "cleanup_preflight";
 const DECOMMISSION_STAGE_SOURCE_CLEANUP: &str = "source_cleanup";
@@ -1832,6 +1834,13 @@ pub(crate) struct PoolRebalanceActivationFence {
 }
 
 impl PoolRebalanceActivationFence {
+    pub(crate) fn set_fleet_proof(
+        &mut self,
+        fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
+    ) {
+        self.fleet_proof = fleet_proof;
+    }
+
     pub(crate) fn ensure_held(&self) -> Result<()> {
         #[cfg(test)]
         let forced_lost = self.forced_lost.load(Ordering::Acquire);
@@ -1845,7 +1854,7 @@ impl PoolRebalanceActivationFence {
             .as_ref()
             .is_some_and(|proof| !crate::services::notification_sys::cross_pool_fence_fleet_proof_matches(proof))
         {
-            return Err(Error::other("pool activation fleet capability proof expired before commit"));
+            return Err(Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED));
         }
 
         Ok(())
@@ -1901,7 +1910,17 @@ pub(crate) async fn acquire_pool_activation_fleet_proof(
     }
     crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof()
         .map(Some)
-        .ok_or_else(|| Error::other("pool activation requires a live fleet capability proof"))
+        .ok_or_else(|| Error::other(POOL_ACTIVATION_FLEET_PROOF_REQUIRED))
+}
+
+pub(crate) fn is_pool_activation_fleet_proof_error(err: &Error) -> bool {
+    // Save-stage helpers add context by formatting the original error, so the
+    // marker may be nested in the display string. Restrict matching to the
+    // `Error::other` I/O shape used by this activation path.
+    matches!(err, Error::Io(io_error) if io_error.kind() == std::io::ErrorKind::Other && {
+        let message = io_error.to_string();
+        message.contains(POOL_ACTIVATION_FLEET_PROOF_REQUIRED) || message.contains(POOL_ACTIVATION_FLEET_PROOF_EXPIRED)
+    })
 }
 
 #[cfg(test)]
@@ -5796,7 +5815,7 @@ fn decommission_remote_tiered_opts(
         versioned: version_id.is_some(),
         version_id,
         mod_time: version.mod_time,
-        user_defined: version.metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect(),
+        user_defined: version.metadata.clone(),
         src_pool_idx,
         data_movement: true,
         incl_free_versions: version.tier_free_version(),
@@ -10828,6 +10847,15 @@ mod tests {
     use crate::bucket::replication::{ReplicationState, ReplicationStatusType};
     use serde::Serialize;
 
+    #[test]
+    fn pool_activation_fleet_proof_error_classifier_matches_only_retryable_proof_failures() {
+        assert!(is_pool_activation_fleet_proof_error(&Error::other(POOL_ACTIVATION_FLEET_PROOF_REQUIRED)));
+        assert!(is_pool_activation_fleet_proof_error(&Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED)));
+        let wrapped = format!("rebalance meta save failed during start_rebalance: {POOL_ACTIVATION_FLEET_PROOF_EXPIRED}");
+        assert!(is_pool_activation_fleet_proof_error(&Error::other(wrapped)));
+        assert!(!is_pool_activation_fleet_proof_error(&Error::ConfigNotFound));
+    }
+
     #[tokio::test]
     #[serial_test::serial]
     async fn decommission_activation_fence_loss_after_durable_save_blocks_publication() {

crates/ecstore/src/error/mod.rs

@@ -157,6 +157,8 @@ pub enum StorageError {
     InvalidPartNumber(usize),
     #[error("Your proposed upload is smaller than the minimum allowed size. Part {0} size {1} is less than minimum {2}")]
     EntityTooSmall(usize, i64, i64),
+    #[error("multipart upload size {0} exceeds the configured limit {1}")]
+    EntityTooLarge(u64, u64),
 
     // ── Erasure / Quorum ─────────────────────────────────────────────
     #[error("erasure read quorum")]
@@ -554,6 +556,7 @@ impl Clone for StorageError {
             StorageError::DecommissionNotStarted => StorageError::DecommissionNotStarted,
             StorageError::InvalidPart(a, b, c) => StorageError::InvalidPart(*a, b.clone(), c.clone()),
             StorageError::EntityTooSmall(a, b, c) => StorageError::EntityTooSmall(*a, *b, *c),
+            StorageError::EntityTooLarge(a, b) => StorageError::EntityTooLarge(*a, *b),
             StorageError::DoneForNow => StorageError::DoneForNow,
             StorageError::DecommissionAlreadyRunning => StorageError::DecommissionAlreadyRunning,
             StorageError::RebalanceAlreadyRunning => StorageError::RebalanceAlreadyRunning,
@@ -673,6 +676,7 @@ impl StorageError {
             StorageError::InsufficientWriteQuorum(_, _) => StorageErrorCode::InsufficientWriteQuorum,
             StorageError::PreconditionFailed => StorageErrorCode::PreconditionFailed,
             StorageError::EntityTooSmall(_, _, _) => StorageErrorCode::EntityTooSmall,
+            StorageError::EntityTooLarge(_, _) => StorageErrorCode::EntityTooLarge,
             StorageError::InvalidRangeSpec(_) => StorageErrorCode::InvalidRangeSpec,
             StorageError::NotModified => StorageErrorCode::NotModified,
             StorageError::InvalidPartNumber(_) => StorageErrorCode::InvalidPartNumber,
@@ -795,6 +799,7 @@ impl StorageError {
             StorageErrorCode::EntityTooSmall => {
                 Some(StorageError::EntityTooSmall(Default::default(), Default::default(), Default::default()))
             }
+            StorageErrorCode::EntityTooLarge => Some(StorageError::EntityTooLarge(Default::default(), Default::default())),
             StorageErrorCode::InvalidRangeSpec => Some(StorageError::InvalidRangeSpec(Default::default())),
             StorageErrorCode::NotModified => Some(StorageError::NotModified),
             StorageErrorCode::InvalidPartNumber => Some(StorageError::InvalidPartNumber(Default::default())),

crates/ecstore/src/object_api/mod.rs

@@ -64,7 +64,7 @@ pub(crate) const ENCRYPTED_FRAME_LAYOUT_FIXED8K_SUFFIX: &str = "encrypted-frame-
 pub(crate) const ENV_RUSTFS_ENCRYPTED_RANGE_SEEK: &str = "RUSTFS_ENCRYPTED_RANGE_SEEK";
 pub(crate) const DEFAULT_RUSTFS_ENCRYPTED_RANGE_SEEK: bool = true;
 
-pub(crate) fn has_encrypted_part_layout_marker<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>, suffix: &str, expected: &str) -> bool {
+pub(crate) fn has_encrypted_part_layout_marker(metadata: &HashMap<String, String>, suffix: &str, expected: &str) -> bool {
     let mut value = None;
     for (key, candidate) in metadata {
         if !rustfs_utils::http::has_internal_suffix(key, suffix) {

crates/ecstore/src/object_api/object_api_utils.rs

@@ -168,7 +168,7 @@ pub fn to_s3s_etag(etag: &str) -> ETag {
     ETag::Strong(etag.to_string())
 }
 
-pub fn get_raw_etag<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> String {
+pub fn get_raw_etag(metadata: &HashMap<String, String>) -> String {
     metadata
         .get("etag")
         .cloned()

crates/ecstore/src/object_api/types.rs

@@ -1008,7 +1008,7 @@ impl ObjectInfo {
             successor_mod_time: fi.successor_mod_time,
             etag,
             inlined,
-            user_defined: Arc::new(metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect()),
+            user_defined: Arc::new(metadata),
             transitioned_object,
             transition_version_state: fi.transition_version_state,
             checksum: fi.checksum.clone(),
@@ -1316,7 +1316,7 @@ impl ObjectInfo {
         if part > 0
             && let Some(checksums) = self.parts.iter().find(|p| p.number == part).and_then(|p| p.checksums.clone())
         {
-            return Ok((checksums.iter().map(|(k, v)| (k.clone(), v.clone())).collect(), true));
+            return Ok((checksums, true));
         }
 
         if let Some(data) = &self.checksum {

crates/ecstore/src/services/notification_sys.rs

@@ -234,6 +234,39 @@ pub(crate) fn install_cross_pool_fence_fleet_proof_for_test() {
     });
 }
 
+#[cfg(test)]
+pub(crate) struct CrossPoolFenceFleetProofGuard {
+    previous_proof: Option<FleetCapabilityProof>,
+    previous_topology_conflict: bool,
+}
+
+#[cfg(test)]
+impl Drop for CrossPoolFenceFleetProofGuard {
+    fn drop(&mut self) {
+        let mut state = cross_pool_fence_fleet_proof_slot()
+            .write()
+            .unwrap_or_else(std::sync::PoisonError::into_inner);
+        state.proof = self.previous_proof.take();
+        state.topology_conflict = self.previous_topology_conflict;
+    }
+}
+
+/// Temporarily revoke the test proof so activation paths can exercise their
+/// fail-closed behavior without changing the process-wide topology binding.
+#[cfg(test)]
+pub(crate) fn without_cross_pool_fence_fleet_proof_for_test() -> CrossPoolFenceFleetProofGuard {
+    let mut state = cross_pool_fence_fleet_proof_slot()
+        .write()
+        .unwrap_or_else(std::sync::PoisonError::into_inner);
+    let guard = CrossPoolFenceFleetProofGuard {
+        previous_proof: state.proof.clone(),
+        previous_topology_conflict: state.topology_conflict,
+    };
+    state.proof = None;
+    state.topology_conflict = true;
+    guard
+}
+
 #[cfg(any(test, feature = "test-util"))]
 pub fn rotate_cross_pool_fence_fleet_proof_for_test() -> bool {
     let mut state = cross_pool_fence_fleet_proof_slot()

crates/ecstore/src/services/rebalance/control.rs

@@ -570,10 +570,13 @@ impl ECStore {
     where
         S: EcstoreObjectIO + StorageNamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
     {
-        let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
+        // Lock order: pool_meta_save_gate -> pool.bin -> rebalance.bin.
         let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
         pool_meta_guard.ensure_write_safe("rebalance worker activation")?;
-        let activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), fleet_proof).await?;
+        // Classify the durable rebalance record while holding both namespace
+        // fences. A terminal record is a no-op and must not depend on the
+        // notification subsystem having published a fleet proof yet.
+        let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?;
         let pool_meta = self
             .load_runtime_pool_meta_under_activation_fence(&mut pool_meta_guard, &activation_fence, "rebalance worker activation")
             .await?;
@@ -597,10 +600,17 @@ impl ECStore {
         }
 
         activation_fence.ensure_held()?;
-        if !is_rebalance_conflicting_with_decommission(&persisted) {
+        if !crate::services::rebalance::rebalance_requires_worker_activation(&persisted) {
             return Ok(RebalanceWorkerActivationFence::NotStartedTerminal);
         }
 
+        // Active worker admission still requires the fail-closed fleet proof.
+        // Attach it immediately before the final fence validation so expiry or
+        // topology changes are checked again at every later commit boundary.
+        let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
+        activation_fence.set_fleet_proof(fleet_proof);
+        activation_fence.ensure_held()?;
+
         Ok(RebalanceWorkerActivationFence::Ready(Box::new(activation_fence)))
     }
 
@@ -1476,6 +1486,64 @@ mod tests {
         assert_activation_locks_released(&store).await;
     }
 
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn rebalance_worker_skips_terminal_metadata_without_fleet_proof() {
+        let rebalance_id = "terminal-metadata-without-proof";
+        let completed = RebalanceMeta {
+            id: rebalance_id.to_string(),
+            pool_stats: vec![RebalanceStats {
+                participating: true,
+                info: RebalanceInfo {
+                    status: RebalStatus::Completed,
+                    ..Default::default()
+                },
+                ..Default::default()
+            }],
+            ..Default::default()
+        };
+        let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(Some(completed)).await;
+        let _proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
+
+        let activation = store
+            .fence_rebalance_worker_activation(store.pools[0].clone(), rebalance_id)
+            .await
+            .expect("terminal metadata should not require a fleet proof");
+        assert!(matches!(activation, RebalanceWorkerActivationFence::NotStartedTerminal));
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn rebalance_worker_still_requires_fleet_proof_for_active_metadata() {
+        let rebalance_id = "active-metadata-without-proof";
+        let active = RebalanceMeta {
+            id: rebalance_id.to_string(),
+            pool_stats: vec![RebalanceStats {
+                participating: true,
+                info: RebalanceInfo {
+                    status: RebalStatus::Started,
+                    ..Default::default()
+                },
+                ..Default::default()
+            }],
+            ..Default::default()
+        };
+        let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(Some(active)).await;
+        let _proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
+
+        let err = match store
+            .fence_rebalance_worker_activation(store.pools[0].clone(), rebalance_id)
+            .await
+        {
+            Ok(_) => panic!("active metadata must not be admitted without a fleet proof"),
+            Err(err) => err,
+        };
+        assert!(
+            err.to_string()
+                .contains("pool activation requires a live fleet capability proof")
+        );
+    }
+
     #[tokio::test]
     #[serial_test::serial]
     async fn rebalance_activation_adopts_commit_after_post_save_fence_loss() {

crates/ecstore/src/services/rebalance/meta.rs

@@ -214,6 +214,14 @@ pub(super) fn is_rebalance_in_progress(meta: &RebalanceMeta) -> bool {
     meta.pool_stats.iter().any(is_rebalance_pool_active)
 }
 
+/// Persisted rebalance metadata requires worker activation only while it has
+/// not reached a durable terminal marker and at least one pool is still marked
+/// active. Merely finding `rebalance.bin` is not evidence that admission is
+/// required: terminal metadata is retained for status reporting.
+pub(crate) fn rebalance_requires_worker_activation(meta: &RebalanceMeta) -> bool {
+    meta.stopped_at.is_none() && is_rebalance_in_progress(meta)
+}
+
 pub(crate) fn is_rebalance_conflicting_with_decommission(meta: &RebalanceMeta) -> bool {
     is_rebalance_in_progress(meta)
 }

crates/ecstore/src/services/rebalance/migration.rs

@@ -57,7 +57,7 @@ fn rebalance_remote_tiered_opts(
         versioned: version_id.is_some(),
         version_id,
         mod_time: version.mod_time,
-        user_defined: version.metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect(),
+        user_defined: version.metadata.clone(),
         src_pool_idx,
         data_movement: true,
         include_part_checksums: true,

crates/ecstore/src/services/rebalance/mod.rs

@@ -49,8 +49,8 @@ mod worker;
 
 #[cfg(feature = "test-util")]
 pub use entry::test_util::PausedRebalanceEntryTestFixture;
-pub(crate) use meta::is_rebalance_conflicting_with_decommission;
 pub use meta::{decode_rebalance_stop_propagation_record, encode_rebalance_stop_propagation_record};
+pub(crate) use meta::{is_rebalance_conflicting_with_decommission, rebalance_requires_worker_activation};
 pub use types::{
     DiskStat, RebalSaveOpt, RebalStatus, RebalanceCleanupWarningEntry, RebalanceCleanupWarnings, RebalanceInfo, RebalanceMeta,
     RebalanceStats, RebalanceStopPropagationRecord,

crates/ecstore/src/services/rebalance/rebalance_unit_tests.rs

@@ -22,11 +22,11 @@ use super::meta::{
     is_rebalance_in_progress, is_rebalance_meta_replaceable_for_new_id, is_rebalance_stopped_terminal_event,
     mark_rebalance_bucket_done, merge_rebalance_bucket_lists, merge_rebalance_meta, next_rebal_bucket_from_stat,
     percent_free_ratio, rebalance_goal_reached, rebalance_meta_load_no_data_error, rebalance_meta_load_unknown_format_error,
-    rebalance_meta_load_unknown_version_error, record_rebalance_cleanup_warning_in_meta, remove_rebalanced_buckets_from_queue,
-    resolve_next_rebalance_bucket, resolve_rebalance_participants, should_accept_rebalance_stats_update,
-    should_ignore_rebalance_data_usage_cache, should_pool_participate, should_preserve_rebalance_stopped_state,
-    should_skip_start_rebalance, stop_rebalance_meta_snapshot, stop_rebalance_state, take_bucket_from_rebalance_queue,
-    validate_init_rebalance_state, validate_start_rebalance_state,
+    rebalance_meta_load_unknown_version_error, rebalance_requires_worker_activation, record_rebalance_cleanup_warning_in_meta,
+    remove_rebalanced_buckets_from_queue, resolve_next_rebalance_bucket, resolve_rebalance_participants,
+    should_accept_rebalance_stats_update, should_ignore_rebalance_data_usage_cache, should_pool_participate,
+    should_preserve_rebalance_stopped_state, should_skip_start_rebalance, stop_rebalance_meta_snapshot, stop_rebalance_state,
+    take_bucket_from_rebalance_queue, validate_init_rebalance_state, validate_start_rebalance_state,
 };
 use super::migration::{
     MigrationBackend, MigrationVersionResult, migrate_entry_version, migrate_entry_version_with_retry_wait,
@@ -3386,6 +3386,52 @@ fn test_is_rebalance_in_progress_only_started_participants() {
     assert!(is_rebalance_in_progress(&meta));
 }
 
+#[test]
+fn test_rebalance_requires_worker_activation_only_for_active_non_stopped_metadata() {
+    let now = OffsetDateTime::now_utc();
+    let active = RebalanceMeta {
+        pool_stats: vec![RebalanceStats {
+            participating: true,
+            info: RebalanceInfo {
+                status: RebalStatus::Started,
+                ..Default::default()
+            },
+            ..Default::default()
+        }],
+        ..Default::default()
+    };
+    let stopped_active = RebalanceMeta {
+        stopped_at: Some(now),
+        pool_stats: active.pool_stats.clone(),
+        ..Default::default()
+    };
+
+    assert!(rebalance_requires_worker_activation(&active));
+    for status in [
+        RebalStatus::Completed,
+        RebalStatus::Stopped,
+        RebalStatus::Failed,
+        RebalStatus::None,
+    ] {
+        let terminal = RebalanceMeta {
+            pool_stats: vec![RebalanceStats {
+                participating: true,
+                info: RebalanceInfo {
+                    status,
+                    ..Default::default()
+                },
+                ..Default::default()
+            }],
+            ..Default::default()
+        };
+        assert!(
+            !rebalance_requires_worker_activation(&terminal),
+            "terminal status {status:?} must not resume"
+        );
+    }
+    assert!(!rebalance_requires_worker_activation(&stopped_active));
+}
+
 #[test]
 fn test_is_rebalance_conflicting_with_decommission_true_when_in_progress() {
     let now = OffsetDateTime::now_utc();

crates/ecstore/src/services/tier/tier.rs

@@ -1454,7 +1454,7 @@ fn tier_backend_identity(config: &TierConfig) -> io::Result<TierDestinationId> {
     encode_tier_backend_identity(tier_type, endpoint, bucket, prefix, region, routing_account)
 }
 
-pub(crate) fn tier_destination_id_from_metadata<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> io::Result<Option<TierDestinationId>> {
+pub(crate) fn tier_destination_id_from_metadata(metadata: &HashMap<String, String>) -> io::Result<Option<TierDestinationId>> {
     let Some(encoded) = rustfs_utils::http::metadata_compat::get_consistent_str(
         metadata,
         rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID,

crates/ecstore/src/set_disk/metadata.rs

@@ -609,7 +609,7 @@ impl SetDisks {
             || Self::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX)
     }
 
-    fn update_hash_quorum_metadata_map<S: std::hash::BuildHasher>(hasher: &mut Sha256, entries: &HashMap<String, String, S>) {
+    fn update_hash_quorum_metadata_map(hasher: &mut Sha256, entries: &HashMap<String, String>) {
         let mut entries = entries
             .iter()
             .filter(|(name, _)| !Self::is_replication_quorum_metadata_key(name))
@@ -635,7 +635,7 @@ impl SetDisks {
     /// so the dual internal prefixes carrying the same mapping share one
     /// identity, while a genuine disagreement between disks still changes the
     /// hash and surfaces as a quorum difference.
-    fn update_hash_target_delete_marker_versions<S: std::hash::BuildHasher>(hasher: &mut Sha256, metadata: &HashMap<String, String, S>) {
+    fn update_hash_target_delete_marker_versions(hasher: &mut Sha256, metadata: &HashMap<String, String>) {
         let (versions, corrupt) = http::target_delete_marker_versions(metadata);
         hasher.update([u8::from(corrupt)]);
         let mut versions = versions.iter().collect::<Vec<_>>();

crates/ecstore/src/set_disk/mod.rs

@@ -190,7 +190,7 @@ use tracing::error;
 use tracing::{Instrument, debug, info, warn};
 use uuid::Uuid;
 
-pub(super) fn restore_operation_id_from_metadata<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> Result<Option<Uuid>> {
+pub(super) fn restore_operation_id_from_metadata(metadata: &HashMap<String, String>) -> Result<Option<Uuid>> {
     let Some(value) = rustfs_utils::http::metadata_compat::get_consistent_str(metadata, SUFFIX_RESTORE_OPERATION_ID) else {
         if rustfs_utils::http::metadata_compat::contains_key_str(metadata, SUFFIX_RESTORE_OPERATION_ID) {
             return Err(Error::other("invalid restore operation id metadata".to_string()));
@@ -204,14 +204,14 @@ pub(super) fn restore_operation_id_from_metadata<S: std::hash::BuildHasher>(meta
     Ok(Some(id))
 }
 
-pub(super) fn require_restore_operation_id<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>, expected: Uuid) -> Result<()> {
+pub(super) fn require_restore_operation_id(metadata: &HashMap<String, String>, expected: Uuid) -> Result<()> {
     match restore_operation_id_from_metadata(metadata)? {
         Some(actual) if actual == expected => Ok(()),
         _ => Err(Error::other("restore operation id changed before copy-back".to_string())),
     }
 }
 
-pub(super) fn restore_commit_operation_id_from_metadata<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> Result<Option<Uuid>> {
+pub(super) fn restore_commit_operation_id_from_metadata(metadata: &HashMap<String, String>) -> Result<Option<Uuid>> {
     if !metadata.contains_key(X_AMZ_RESTORE.as_str()) {
         return Ok(None);
     }
@@ -466,13 +466,13 @@ fn release_materialized_read_lock(bucket: &str, object: &str, read_lock_guard: O
     drop(read_lock_guard);
 }
 
-pub(crate) fn strip_internal_multipart_metadata<S: std::hash::BuildHasher>(metadata: &mut HashMap<String, String, S>) {
+pub(crate) fn strip_internal_multipart_metadata(metadata: &mut HashMap<String, String>) {
     metadata.remove(RUSTFS_MULTIPART_BUCKET_KEY);
     metadata.remove(RUSTFS_MULTIPART_OBJECT_KEY);
     rustfs_utils::http::metadata_compat::remove_str(metadata, SUFFIX_BUCKET_INCARNATION_ID);
 }
 
-fn should_persist_encryption_original_size<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> bool {
+fn should_persist_encryption_original_size(metadata: &HashMap<String, String>) -> bool {
     metadata.keys().any(|key| is_object_encryption_marker(key))
 }
 

crates/ecstore/src/set_disk/ops/multipart.rs

@@ -84,11 +84,13 @@ use rustfs_rio::TryGetIndex;
 use rustfs_utils::http::SSEC_ALGORITHM_HEADER;
 #[cfg(test)]
 use rustfs_utils::http::SUFFIX_COMPRESSION;
+use rustfs_utils::http::{SUFFIX_MAX_TOTAL_OBJECT_SIZE, get_consistent_str};
 use std::future::Future;
 #[cfg(test)]
 use std::sync::atomic::AtomicBool;
 #[cfg(any(test, feature = "test-util"))]
 use std::sync::atomic::{AtomicUsize, Ordering};
+use std::sync::{Mutex, OnceLock};
 #[cfg(any(test, feature = "test-util"))]
 use std::time::Duration;
 #[cfg(test)]
@@ -97,6 +99,83 @@ use tokio::task::JoinSet;
 
 const MULTIPART_LIST_IO_CONCURRENCY: usize = 16;
 
+static CAPPED_MULTIPART_STAGING: OnceLock<Mutex<HashMap<String, Arc<tokio::sync::Semaphore>>>> = OnceLock::new();
+
+struct CappedMultipartStagingGuard {
+    upload_id_path: String,
+    permit: Option<tokio::sync::OwnedSemaphorePermit>,
+}
+
+impl Drop for CappedMultipartStagingGuard {
+    fn drop(&mut self) {
+        // Release the permit before checking the Arc count so a concurrent
+        // Abort/Complete cleanup can remove the now-unused map entry.
+        self.permit.take();
+        remove_capped_multipart_staging_semaphore(&self.upload_id_path);
+    }
+}
+
+fn capped_multipart_staging_semaphore(upload_id_path: &str) -> Arc<tokio::sync::Semaphore> {
+    CAPPED_MULTIPART_STAGING
+        .get_or_init(|| Mutex::new(HashMap::new()))
+        .lock()
+        .expect("capped multipart staging semaphore map should not be poisoned")
+        .entry(upload_id_path.to_owned())
+        .or_insert_with(|| Arc::new(tokio::sync::Semaphore::new(1)))
+        .clone()
+}
+
+fn remove_capped_multipart_staging_semaphore(upload_id_path: &str) {
+    if let Some(map) = CAPPED_MULTIPART_STAGING.get() {
+        let mut map = map
+            .lock()
+            .expect("capped multipart staging semaphore map should not be poisoned");
+        let removable = map
+            .get(upload_id_path)
+            .is_some_and(|semaphore| Arc::strong_count(semaphore) == 1);
+        if removable {
+            map.remove(upload_id_path);
+        }
+    }
+}
+
+fn multipart_size_limit_from_metadata(metadata: &HashMap<String, String>) -> Result<Option<u64>> {
+    if !contains_key_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE) {
+        return Ok(None);
+    }
+
+    let Some(value) = get_consistent_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE) else {
+        return Err(Error::InvalidArgument(
+            "multipart upload".to_string(),
+            SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
+            "missing or conflicting internal size limit".to_string(),
+        ));
+    };
+
+    let limit = value.parse::<u64>().map_err(|_| {
+        Error::InvalidArgument(
+            "multipart upload".to_string(),
+            SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
+            "invalid internal size limit".to_string(),
+        )
+    })?;
+    Ok(Some(limit))
+}
+
+fn admitted_multipart_size(current: u64, candidate: u64, limit: u64) -> Result<u64> {
+    let total = current.checked_add(candidate).ok_or_else(|| {
+        Error::InvalidArgument(
+            "multipart upload".to_string(),
+            SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
+            "logical size overflow".to_string(),
+        )
+    })?;
+    if total > limit {
+        return Err(Error::EntityTooLarge(total, limit));
+    }
+    Ok(total)
+}
+
 pub(crate) struct StaleMultipartCleanupGuard {
     file_info: FileInfo,
     upload_path: String,
@@ -115,8 +194,13 @@ impl StaleMultipartCleanupGuard {
 
     pub(crate) async fn delete(self, set: &SetDisks) -> Result<()> {
         fence_commit_on_lock_loss(Some(&self.lock_guard), "stale_multipart_cleanup", &self.upload_path)?;
-        set.delete_all_with_quorum(RUSTFS_META_MULTIPART_BUCKET, &self.upload_path, self.write_quorum)
-            .await
+        let result = set
+            .delete_all_with_quorum(RUSTFS_META_MULTIPART_BUCKET, &self.upload_path, self.write_quorum)
+            .await;
+        if result.is_ok() {
+            remove_capped_multipart_staging_semaphore(&self.upload_path);
+        }
+        result
     }
 }
 
@@ -365,7 +449,7 @@ fn fence_commit_on_lock_loss(guard: Option<&ObjectLockDiagGuard>, mode: &'static
     Ok(())
 }
 
-fn multipart_bucket_incarnation_id<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> Result<Option<Uuid>> {
+fn multipart_bucket_incarnation_id(metadata: &HashMap<String, String>) -> Result<Option<Uuid>> {
     let Some(value) = rustfs_utils::http::metadata_compat::get_consistent_str(metadata, SUFFIX_BUCKET_INCARNATION_ID) else {
         if rustfs_utils::http::metadata_compat::contains_key_str(metadata, SUFFIX_BUCKET_INCARNATION_ID) {
             return Err(Error::other("invalid multipart bucket incarnation metadata"));
@@ -379,12 +463,12 @@ fn multipart_bucket_incarnation_id<S: std::hash::BuildHasher>(metadata: &HashMap
     Ok(Some(incarnation))
 }
 
-fn multipart_bucket_incarnation_matches<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>, expected: Uuid) -> bool {
+fn multipart_bucket_incarnation_matches(metadata: &HashMap<String, String>, expected: Uuid) -> bool {
     matches!(multipart_bucket_incarnation_id(metadata), Ok(Some(actual)) if actual == expected)
 }
 
-fn validate_multipart_bucket_incarnation<S: std::hash::BuildHasher>(
-    metadata: &HashMap<String, String, S>,
+fn validate_multipart_bucket_incarnation(
+    metadata: &HashMap<String, String>,
     bucket: &str,
     object: &str,
     upload_id: &str,
@@ -635,6 +719,61 @@ async fn multipart_upload_paths_on_disk(disk: DiskStore, bucket: &str) -> disk::
 }
 
 impl SetDisks {
+    async fn current_multipart_logical_size(
+        &self,
+        bucket: &str,
+        object: &str,
+        upload_id: &str,
+        upload_id_path: &str,
+        fi: &FileInfo,
+        replacing_part: usize,
+    ) -> Result<u64> {
+        let online_disks = self.get_disks_internal().await;
+        let read_quorum = fi.read_quorum(self.default_read_quorum());
+        let part_path = format!(
+            "{}{}",
+            path_join_buf(&[
+                upload_id_path,
+                fi.data_dir.map(|v| v.to_string()).unwrap_or_default().as_str(),
+            ]),
+            SLASH_SEPARATOR
+        );
+        let part_numbers = match Self::list_parts(&online_disks, &part_path, read_quorum).await {
+            Ok(parts) => parts,
+            Err(DiskError::FileNotFound) => return Ok(0),
+            Err(err) => return Err(to_object_err(err.into(), vec![bucket, object, upload_id])),
+        };
+        if part_numbers.is_empty() {
+            return Ok(0);
+        }
+        let part_meta_paths = part_numbers
+            .iter()
+            .map(|number| format!("{part_path}part.{number}.meta"))
+            .collect::<Vec<_>>();
+        let existing_parts =
+            Self::read_parts(&online_disks, RUSTFS_META_MULTIPART_BUCKET, &part_meta_paths, &part_numbers, read_quorum)
+                .await
+                .map_err(|err| to_object_err(err.into(), vec![bucket, object, upload_id]))?;
+
+        existing_parts.into_iter().try_fold(0_u64, |total, part| {
+            if part.error.is_some() || part.number == replacing_part {
+                return if part.error.is_some() {
+                    Err(Error::PartMissingOrCorrupt)
+                } else {
+                    Ok(total)
+                };
+            }
+            let part_size = u64::try_from(part.actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
+            total.checked_add(part_size).ok_or_else(|| {
+                Error::InvalidArgument(
+                    "multipart upload".to_string(),
+                    SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
+                    "logical size overflow".to_string(),
+                )
+            })
+        })
+    }
+
     async fn discover_multipart_upload_paths(
         &self,
         orig_bucket: &str,
@@ -1130,9 +1269,18 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
         crate::hp_guard!("SetDisks::put_object_part");
         let upload_id_path = Self::get_multipart_upload_dir(bucket, object, upload_id, opts.data_movement);
 
-        let (fi, _) = self
+        let (fi, _) = match self
             .check_upload_id_exists_with_opts(bucket, object, upload_id, true, opts)
-            .await?;
+            .await
+        {
+            Ok(value) => value,
+            Err(err @ Error::InvalidUploadID(..)) => {
+                remove_capped_multipart_staging_semaphore(&upload_id_path);
+                return Err(err);
+            }
+            Err(err) => return Err(err),
+        };
+        let multipart_size_limit = multipart_size_limit_from_metadata(&fi.metadata)?;
         ensure_data_movement_upload_access(&fi, bucket, object, upload_id, opts)?;
         ensure_multipart_bucket_incarnation(&self.ctx, &fi, bucket, object, upload_id, opts.expected_bucket_incarnation_id)
             .await?;
@@ -1165,6 +1313,44 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
         let part_suffix = format!("part.{part_id}");
         let tmp_part = format!("{}x{}", Uuid::new_v4(), OffsetDateTime::now_utc().unix_timestamp());
         let tmp_part_path = Arc::new(format!("{tmp_part}/{part_suffix}"));
+        let part_path = format!("{}/{}/{}", upload_id_path, fi.data_dir.unwrap_or_default(), part_suffix);
+        let part_lock_path = format!("{upload_id_path}/{part_suffix}");
+
+        // Keep at most one capped part staging locally per upload. The
+        // distributed lock below is held only for the durable admission check;
+        // it is reacquired for the short final rename, so Complete/Abort are
+        // not blocked behind a slow body upload.
+        let _capped_staging_guard = if multipart_size_limit.is_some() {
+            Some(CappedMultipartStagingGuard {
+                upload_id_path: upload_id_path.clone(),
+                permit: Some(
+                    capped_multipart_staging_semaphore(&upload_id_path)
+                        .acquire_owned()
+                        .await
+                        .map_err(|_| Error::other("capped multipart staging semaphore closed"))?,
+                ),
+            })
+        } else {
+            None
+        };
+
+        if let Some(limit) = multipart_size_limit {
+            let admission_guard = self
+                .acquire_write_lock_diag("put_object_part_admission", RUSTFS_META_MULTIPART_BUCKET, &upload_id_path)
+                .await?;
+            let declared_size = if data.size() >= 0 {
+                u64::try_from(data.size()).map_err(|_| Error::PartMissingOrCorrupt)?
+            } else if data.actual_size() >= 0 {
+                u64::try_from(data.actual_size()).map_err(|_| Error::PartMissingOrCorrupt)?
+            } else {
+                return Err(Error::PartMissingOrCorrupt);
+            };
+            let current_size = self
+                .current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &fi, part_id)
+                .await?;
+            admitted_multipart_size(current_size, declared_size, limit)?;
+            drop(admission_guard);
+        }
 
         let result: Result<PartInfo> = async {
             let erasure =
@@ -1335,7 +1521,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
                 mod_time: Some(OffsetDateTime::now_utc()),
                 actual_size,
                 index: index_op,
-                checksums: if checksums.is_empty() { None } else { Some(checksums.iter().map(|(k, v)| (k.clone(), v.clone())).collect()) },
+                checksums: if checksums.is_empty() { None } else { Some(checksums) },
                 ..Default::default()
             };
 
@@ -1365,30 +1551,21 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
                 .await?;
             }
 
-            let part_path = format!("{}/{}/{}", upload_id_path, fi.data_dir.unwrap_or_default(), part_suffix);
-            let part_lock_path = format!("{upload_id_path}/{part_suffix}");
-
             #[cfg(test)]
             pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockAcquire).await;
-            // Serialize only same-part commits (rename_part), not the whole upload.
-            // Each concurrent stream writes to its own unique temp dir (see
-            // `tmp_part` above), so the encode/stream phase never conflicts and must
-            // stay lock-free — holding a lock across it would serialize slow
-            // re-transmits of the same part and defeat the S3 "last finisher wins"
-            // semantics. The mixed-generation hazard is confined to rename_part,
-            // where two temp parts are moved cross-disk onto the SAME final
-            // part_path: interleaving there can leave shards from two generations,
-            // each individually bitrot-valid, that only surface as silent corruption
-            // at read time (backlog#853). A write lock scoped to this part number
-            // makes each same-part commit atomic across disks, so the last committer
-            // wins consistently, while different part numbers commit onto disjoint
-            // part paths and stay concurrent (issue#5961 — an uploadId-wide write
-            // lock serialized them into 503 lock-acquire timeouts). The shared
-            // uploadId read lock keeps completion/abort (which take the uploadId
-            // write lock) from racing any in-flight part commit; a guarded
-            // completion takes the object lock before the upload lock to preserve
-            // global ordering.
-            let (_upload_commit_guard, _part_commit_guard) = if opts.no_lock {
+            // Capped uploads reacquire the upload-wide write lock for the
+            // final durable check and rename. Uncapped uploads retain the
+            // concurrent encode path and only serialize the final same-part
+            // rename; completion/abort use the upload-wide write lock.
+            let (_upload_commit_guard, _part_commit_guard) = if multipart_size_limit.is_some() {
+                let upload_guard = self
+                    .acquire_write_lock_diag("put_object_part_commit", RUSTFS_META_MULTIPART_BUCKET, &upload_id_path)
+                    .await?;
+                let part_guard = self
+                    .acquire_write_lock_diag("put_object_part_commit", RUSTFS_META_MULTIPART_BUCKET, &part_lock_path)
+                    .await?;
+                (Some(upload_guard), Some(part_guard))
+            } else if opts.no_lock {
                 (None, None)
             } else {
                 let upload_guard = self
@@ -1400,8 +1577,16 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
                 (Some(upload_guard), Some(part_guard))
             };
             let (commit_fi, _) = self
-                .check_upload_id_exists_with_opts(bucket, object, upload_id, false, opts)
+                .check_upload_id_exists_with_opts(bucket, object, upload_id, multipart_size_limit.is_some(), opts)
                 .await?;
+            let commit_size_limit = multipart_size_limit_from_metadata(&commit_fi.metadata)?;
+            if commit_size_limit != multipart_size_limit {
+                return Err(Error::InvalidArgument(
+                    "multipart upload".to_string(),
+                    SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
+                    "size limit metadata changed or is missing".to_string(),
+                ));
+            }
             ensure_data_movement_upload_access(&commit_fi, bucket, object, upload_id, opts)?;
             ensure_multipart_bucket_incarnation(
                 &self.ctx,
@@ -1431,6 +1616,14 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
             }
             ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
 
+            if let Some(limit) = commit_size_limit {
+                let current_size = self
+                    .current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &commit_fi, part_id)
+                    .await?;
+                let candidate_size = u64::try_from(actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
+                admitted_multipart_size(current_size, candidate_size, limit)?;
+            }
+
             let _ = self
                 .rename_part(
                     &shuffle_disks,
@@ -1523,7 +1716,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
             max_parts,
             part_number_marker,
             user_defined: {
-                let mut metadata: HashMap<String, String> = fi.metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
+                let mut metadata = fi.metadata.clone();
                 strip_internal_multipart_metadata(&mut metadata);
                 metadata
             },
@@ -1782,7 +1975,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
         let mod_time = opts.mod_time.unwrap_or_else(OffsetDateTime::now_utc);
 
         for f in parts_metadatas.iter_mut() {
-            f.metadata = user_defined.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
+            f.metadata = user_defined.clone();
             f.mod_time = Some(mod_time);
             f.fresh = true;
         }
@@ -1871,7 +2064,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
             upload_id: upload_id.to_owned(),
             user_defined: {
                 strip_internal_multipart_metadata(&mut fi.metadata);
-                fi.metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect()
+                fi.metadata.clone()
             },
             ..Default::default()
         })
@@ -1891,12 +2084,17 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
         ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
         let upload_id_path = Self::get_multipart_upload_dir(bucket, object, upload_id, opts.data_movement);
 
-        self.delete_all_with_quorum(
-            RUSTFS_META_MULTIPART_BUCKET,
-            &upload_id_path,
-            fi.write_quorum(self.default_write_quorum()),
-        )
-        .await
+        let result = self
+            .delete_all_with_quorum(
+                RUSTFS_META_MULTIPART_BUCKET,
+                &upload_id_path,
+                fi.write_quorum(self.default_write_quorum()),
+            )
+            .await;
+        if result.is_ok() {
+            remove_capped_multipart_staging_semaphore(&upload_id_path);
+        }
+        result
     }
     // complete_multipart_upload finished
     #[tracing::instrument(skip(self))]
@@ -2016,6 +2214,27 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
             return Err(Error::other("part result number err"));
         }
 
+        if let Some(limit) = multipart_size_limit_from_metadata(&fi.metadata)? {
+            let mut total = 0_u64;
+            for part in &object_parts {
+                if part.error.is_some() {
+                    return Err(Error::PartMissingOrCorrupt);
+                }
+                let part_size = u64::try_from(part.actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
+                total = total.checked_add(part_size).ok_or_else(|| {
+                    Error::InvalidArgument(
+                        "multipart upload".to_string(),
+                        SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
+                        "logical size overflow".to_string(),
+                    )
+                })?;
+            }
+            if total > limit {
+                return Err(Error::EntityTooLarge(total, limit));
+            }
+            rustfs_utils::http::metadata_compat::remove_str(&mut fi.metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE);
+        }
+
         let mut checksum_type = rustfs_rio::ChecksumType::NONE;
 
         if let Some(cs) = fi.metadata.get(rustfs_rio::RUSTFS_MULTIPART_CHECKSUM) {
@@ -3024,13 +3243,17 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
             Ok(ObjectInfo::from_file_info(&fi, &commit_bucket, &commit_object, commit_is_versioned))
         };
 
-        if detach_commit_owner {
+        let result = if detach_commit_owner {
             tokio::spawn(commit)
                 .await
                 .map_err(|err| Error::other(format!("complete_multipart_upload commit task failed: {err}")))?
         } else {
             commit.await
+        };
+        if result.is_ok() {
+            remove_capped_multipart_staging_semaphore(&upload_id_path);
         }
+        result
     }
 }
 
@@ -3096,6 +3319,34 @@ mod tests {
         assert!(multipart_bucket_incarnation_id(&nil_metadata).is_err());
     }
 
+    #[test]
+    fn multipart_size_limit_metadata_is_dual_key_and_fail_closed() {
+        let mut metadata = HashMap::new();
+        insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "100".to_string());
+        assert_eq!(multipart_size_limit_from_metadata(&metadata).unwrap(), Some(100));
+
+        metadata.insert("x-minio-internal-max-total-object-size".to_string(), "101".to_string());
+        assert!(multipart_size_limit_from_metadata(&metadata).is_err());
+
+        let mut invalid = HashMap::new();
+        insert_str(&mut invalid, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "-1".to_string());
+        assert!(multipart_size_limit_from_metadata(&invalid).is_err());
+    }
+
+    #[test]
+    fn multipart_size_admission_handles_boundaries_and_overflow() {
+        assert_eq!(admitted_multipart_size(90, 10, 100).unwrap(), 100);
+        assert!(matches!(
+            admitted_multipart_size(90, 11, 100),
+            Err(StorageError::EntityTooLarge(101, 100))
+        ));
+        assert!(admitted_multipart_size(u64::MAX - 1, 1, u64::MAX).is_ok());
+        assert!(matches!(
+            admitted_multipart_size(u64::MAX, 1, u64::MAX),
+            Err(StorageError::InvalidArgument(_, _, _))
+        ));
+    }
+
     #[test]
     fn multipart_bucket_incarnation_gate_accepts_only_current_or_same_lifetime_legacy_uploads() {
         let expected = Uuid::new_v4();

crates/ecstore/src/set_disk/ops/object.rs

@@ -19,7 +19,6 @@
 //! bounds are unchanged, and the impls reach shared primitives through the
 //! SetDisks core (io_primitives) via inherent calls.
 
-use ahash::AHashMap;
 #[cfg(test)]
 use super::super::MetadataCacheInvalidationProbe;
 use super::super::{
@@ -1108,13 +1107,13 @@ fn is_restore_control_metadata(key: &str) -> bool {
             .is_some_and(|remainder| remainder.is_empty())
 }
 
-fn restore_metadata_update_preserves_protected_metadata<S1: std::hash::BuildHasher, S2: std::hash::BuildHasher>(
-    existing: &HashMap<String, String, S1>,
-    replacement: &HashMap<String, String, S2>,
+fn restore_metadata_update_preserves_protected_metadata(
+    existing: &HashMap<String, String>,
+    replacement: &HashMap<String, String>,
 ) -> bool {
-    let mut existing: HashMap<String, String> = existing.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
+    let mut existing = existing.clone();
     clean_metadata(&mut existing);
-    let mut replacement: HashMap<String, String> = replacement.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
+    let mut replacement = replacement.clone();
     clean_metadata(&mut replacement);
     let existing_count = existing.keys().filter(|key| !is_restore_control_metadata(key)).count();
     let replacement_count = replacement.keys().filter(|key| !is_restore_control_metadata(key)).count();
@@ -2212,9 +2211,9 @@ pub(in crate::set_disk) fn stored_replication_category_metadata(existing: &Objec
 ///
 /// Returns whether `inbound` was modified. Callers must hold the object write
 /// lock so the stored values compared here are the ones being replaced.
-pub(in crate::set_disk) fn merge_replication_metadata_lww<S1: std::hash::BuildHasher, S2: std::hash::BuildHasher>(
-    inbound: &mut HashMap<String, String, S1>,
-    existing: &HashMap<String, String, S2>,
+pub(in crate::set_disk) fn merge_replication_metadata_lww(
+    inbound: &mut HashMap<String, String>,
+    existing: &HashMap<String, String>,
     opts: &ObjectOptions,
 ) -> bool {
     use rustfs_utils::http::headers::{
@@ -2845,7 +2844,7 @@ impl SetDisks {
                 )));
             }
 
-            fi.metadata = user_defined.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
+            fi.metadata = user_defined;
             fi.mod_time = mod_time;
             fi.size = w_size as i64;
             fi.versioned = opts.versioned || opts.version_suspended;
@@ -6052,7 +6051,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
         } else {
             None
         };
-        let mut replacement_metadata: AHashMap<String, String> = (*src_info.user_defined).iter().map(|(k, v)| (k.clone(), v.clone())).collect();
+        let mut replacement_metadata = (*src_info.user_defined).clone();
         if let Some(part_checksums) = preserved_part_checksums {
             rustfs_utils::http::insert_str(&mut replacement_metadata, rustfs_utils::http::SUFFIX_PART_CHECKSUMS, part_checksums);
         }
@@ -7494,7 +7493,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
             X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str(),
             X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str(),
         ] {
-            if let Some(value) = fi.metadata.get(header).filter(|value| !value.is_empty()) {
+            if let Some(value) = fi.metadata.lookup(header).filter(|value| !value.is_empty()) {
                 transition_meta.insert(header.to_ascii_lowercase(), value.to_string());
             }
         }

crates/ecstore/src/store/init.rs

@@ -99,6 +99,8 @@ fn preflight_startup_rpc_secret_with(
 const LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES: usize = 6;
 const LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY: Duration = Duration::from_secs(60 * 3);
 const LOCAL_DECOMMISSION_RESUME_RETRY_DELAY: Duration = Duration::from_secs(30);
+const REBALANCE_INITIAL_RESUME_DELAY: Duration = Duration::from_secs(10);
+const REBALANCE_RESUME_RETRY_DELAY: Duration = Duration::from_secs(10);
 
 fn should_retry_local_decommission_resume(err: &Error, attempt: usize) -> bool {
     matches!(err, Error::ConfigNotFound) && attempt < LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES
@@ -108,8 +110,12 @@ fn should_retry_format_load(err: &Error) -> bool {
     !matches!(err, Error::CorruptedFormat)
 }
 
-fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_meta_loaded: bool) -> bool {
-    rebalance_meta_loaded && !decommission_running
+fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_resume_required: bool) -> bool {
+    rebalance_resume_required && !decommission_running
+}
+
+fn should_defer_rebalance_auto_start(distributed: bool, fleet_proof_available: bool) -> bool {
+    distributed && !fleet_proof_available
 }
 
 fn should_schedule_local_decommission_resume(
@@ -127,6 +133,17 @@ async fn wait_for_local_decommission_resume_delay(rx: &CancellationToken, delay:
     }
 }
 
+async fn wait_for_rebalance_resume_delay(rx: &CancellationToken, delay: Duration) -> bool {
+    tokio::select! {
+        _ = rx.cancelled() => false,
+        _ = tokio::time::sleep(delay) => true,
+    }
+}
+
+async fn wait_for_rebalance_resume_retry(rx: &CancellationToken) -> bool {
+    wait_for_rebalance_resume_delay(rx, REBALANCE_RESUME_RETRY_DELAY).await
+}
+
 fn resolve_store_init_stage_result(result: Result<()>, stage: &str) -> Result<()> {
     result.map_err(|err| Error::other(format!("store init failed during {stage}: {err}")))
 }
@@ -283,6 +300,71 @@ async fn resume_local_decommission_after_init(store: Arc<ECStore>, rx: Cancellat
     }
 }
 
+async fn resume_rebalance_after_init(store: Arc<ECStore>, rx: CancellationToken) {
+    if !wait_for_rebalance_resume_delay(&rx, REBALANCE_INITIAL_RESUME_DELAY).await {
+        return;
+    }
+
+    loop {
+        if rx.is_cancelled() {
+            return;
+        }
+
+        let resume_required = store
+            .rebalance_meta
+            .read()
+            .await
+            .as_ref()
+            .is_some_and(crate::services::rebalance::rebalance_requires_worker_activation);
+        if !resume_required {
+            return;
+        }
+
+        if should_defer_rebalance_auto_start(
+            store.ctx.is_dist_erasure().await,
+            crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_some(),
+        ) {
+            if !wait_for_rebalance_resume_retry(&rx).await {
+                return;
+            }
+            continue;
+        }
+
+        match store.start_rebalance().await {
+            Ok(()) => return,
+            Err(err) if crate::core::pools::is_pool_activation_fleet_proof_error(&err) => {
+                warn!(
+                    event = EVENT_ECSTORE_INIT_STATUS,
+                    component = LOG_COMPONENT_ECSTORE,
+                    subsystem = LOG_SUBSYSTEM_STORE_INIT,
+                    stage = "start_rebalance",
+                    state = "retrying",
+                    reason = "fleet_capability_proof_unavailable",
+                    error = %err,
+                    retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
+                    "Retrying deferred rebalance auto-start"
+                );
+                if !wait_for_rebalance_resume_retry(&rx).await {
+                    return;
+                }
+            }
+            Err(err) => {
+                error!(
+                    event = EVENT_ECSTORE_INIT_STATUS,
+                    component = LOG_COMPONENT_ECSTORE,
+                    subsystem = LOG_SUBSYSTEM_STORE_INIT,
+                    stage = "start_rebalance",
+                    state = "failed",
+                    reason = "deferred_resume_failed",
+                    error = %err,
+                    "Failed to resume rebalance after store initialization"
+                );
+                return;
+            }
+        }
+    }
+}
+
 impl ECStore {
     /// Validate topology and process storage-class overrides before any disk is opened.
     pub fn validate_startup_storage_class(endpoint_pools: &EndpointServerPools) -> Result<()> {
@@ -574,12 +656,49 @@ impl ECStore {
         }
 
         resolve_store_init_stage_result(self.load_rebalance_meta().await, "load_rebalance_meta")?;
-        let rebalance_meta_loaded = self.rebalance_meta.read().await.is_some();
+        let rebalance_resume_required = {
+            let rebalance_meta = self.rebalance_meta.read().await;
+            rebalance_meta
+                .as_ref()
+                .is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
+        };
         let decommission_running =
             pool_meta_has_active_decommission(&installed_pool_meta) || self.is_decommission_running().await;
-        if should_auto_start_rebalance_after_init(decommission_running, rebalance_meta_loaded) {
-            resolve_store_init_stage_result(self.start_rebalance().await, "start_rebalance")?;
-        } else if decommission_running && rebalance_meta_loaded {
+        let distributed = self.ctx.is_dist_erasure().await;
+        let fleet_proof_available = crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_some();
+        let mut rebalance_auto_start_deferred = false;
+        if should_auto_start_rebalance_after_init(decommission_running, rebalance_resume_required) {
+            if should_defer_rebalance_auto_start(distributed, fleet_proof_available) {
+                rebalance_auto_start_deferred = true;
+                warn!(
+                    event = EVENT_ECSTORE_INIT_STATUS,
+                    component = LOG_COMPONENT_ECSTORE,
+                    subsystem = LOG_SUBSYSTEM_STORE_INIT,
+                    stage = "start_rebalance",
+                    state = "deferred",
+                    reason = "fleet_capability_proof_unavailable",
+                    retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
+                    "Deferred rebalance auto-start until a live fleet capability proof is available"
+                );
+            } else if let Err(err) = self.start_rebalance().await {
+                if crate::core::pools::is_pool_activation_fleet_proof_error(&err) {
+                    rebalance_auto_start_deferred = true;
+                    warn!(
+                        event = EVENT_ECSTORE_INIT_STATUS,
+                        component = LOG_COMPONENT_ECSTORE,
+                        subsystem = LOG_SUBSYSTEM_STORE_INIT,
+                        stage = "start_rebalance",
+                        state = "deferred",
+                        reason = "fleet_capability_proof_changed",
+                        error = %err,
+                        retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
+                        "Deferred rebalance auto-start after the fleet capability proof changed"
+                    );
+                } else {
+                    return resolve_store_init_stage_result(Err(err), "start_rebalance");
+                }
+            }
+        } else if decommission_running && rebalance_resume_required {
             warn!(
                 event = EVENT_ECSTORE_INIT_STATUS,
                 component = LOG_COMPONENT_ECSTORE,
@@ -616,12 +735,13 @@ impl ECStore {
             .is_ok();
         if should_schedule_local_decommission_resume(&local_pool_indices, pool_meta_replica_state, pool_meta_write_safe) {
             let store = self.clone();
+            let decommission_rx = rx.clone();
 
             tokio::spawn(async move {
-                if !wait_for_local_decommission_resume_delay(&rx, LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY).await {
+                if !wait_for_local_decommission_resume_delay(&decommission_rx, LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY).await {
                     return;
                 }
-                resume_local_decommission_after_init(store, rx, local_pool_indices).await;
+                resume_local_decommission_after_init(store, decommission_rx, local_pool_indices).await;
             });
         } else if !local_pool_indices.is_empty() {
             error!(
@@ -648,6 +768,11 @@ impl ECStore {
             info!("TierConfigMgr init error: {}", err);
         }
 
+        if rebalance_auto_start_deferred {
+            let store = self.clone();
+            tokio::spawn(resume_rebalance_after_init(store, rx));
+        }
+
         Ok(())
     }
 
@@ -665,7 +790,8 @@ mod tests {
         load_pool_meta_for_startup, persist_pool_meta_for_startup_if_safe, pool_first_endpoint_is_local,
         pool_meta_has_active_decommission, preflight_startup_rpc_secret_with, resolve_startup_pool_defaults_with,
         resolve_store_init_stage_result, save_validated_pool_meta_for_startup, should_auto_start_rebalance_after_init,
-        should_retry_format_load, should_retry_local_decommission_resume, wait_for_local_decommission_resume_delay,
+        should_defer_rebalance_auto_start, should_retry_format_load, should_retry_local_decommission_resume,
+        wait_for_local_decommission_resume_delay,
     };
     #[cfg(feature = "test-util")]
     use crate::disk::DiskAPI;
@@ -1450,7 +1576,7 @@ mod tests {
     }
 
     #[test]
-    fn test_should_auto_start_rebalance_after_init_allows_loaded_rebalance_without_decommission() {
+    fn test_should_auto_start_rebalance_after_init_allows_active_rebalance_without_decommission() {
         assert!(should_auto_start_rebalance_after_init(false, true));
     }
 
@@ -1460,10 +1586,17 @@ mod tests {
     }
 
     #[test]
-    fn test_should_auto_start_rebalance_after_init_rejects_missing_rebalance_meta() {
+    fn test_should_auto_start_rebalance_after_init_rejects_terminal_or_missing_rebalance() {
         assert!(!should_auto_start_rebalance_after_init(false, false));
     }
 
+    #[test]
+    fn test_should_defer_rebalance_auto_start_only_without_distributed_fleet_proof() {
+        assert!(should_defer_rebalance_auto_start(true, false));
+        assert!(!should_defer_rebalance_auto_start(true, true));
+        assert!(!should_defer_rebalance_auto_start(false, false));
+    }
+
     #[test]
     fn test_store_init_recovery_skips_rebalance_when_decommission_metadata_is_active() {
         let pool_meta = init_test_pool_meta(Some(PoolDecommissionInfo {
@@ -1473,22 +1606,69 @@ mod tests {
             canceled: false,
             ..Default::default()
         }));
-        let rebalance_meta = Some(RebalanceMeta::default());
+        let rebalance_meta = Some(RebalanceMeta {
+            pool_stats: vec![RebalanceStats {
+                participating: true,
+                info: RebalanceInfo {
+                    status: RebalStatus::Started,
+                    ..Default::default()
+                },
+                ..Default::default()
+            }],
+            ..Default::default()
+        });
 
         assert!(!should_auto_start_rebalance_after_init(
             pool_meta_has_active_decommission(&pool_meta),
-            rebalance_meta.is_some()
+            rebalance_meta
+                .as_ref()
+                .is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
         ));
     }
 
     #[test]
-    fn test_store_init_recovery_allows_rebalance_when_only_rebalance_metadata_exists() {
+    fn test_store_init_recovery_allows_active_rebalance_without_decommission() {
         let pool_meta = init_test_pool_meta(None);
-        let rebalance_meta = Some(RebalanceMeta::default());
+        let rebalance_meta = Some(RebalanceMeta {
+            pool_stats: vec![RebalanceStats {
+                participating: true,
+                info: RebalanceInfo {
+                    status: RebalStatus::Started,
+                    ..Default::default()
+                },
+                ..Default::default()
+            }],
+            ..Default::default()
+        });
 
         assert!(should_auto_start_rebalance_after_init(
             pool_meta_has_active_decommission(&pool_meta),
-            rebalance_meta.is_some()
+            rebalance_meta
+                .as_ref()
+                .is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
+        ));
+    }
+
+    #[test]
+    fn test_store_init_recovery_skips_completed_rebalance_metadata() {
+        let pool_meta = init_test_pool_meta(None);
+        let rebalance_meta = Some(RebalanceMeta {
+            pool_stats: vec![RebalanceStats {
+                participating: true,
+                info: RebalanceInfo {
+                    status: RebalStatus::Completed,
+                    ..Default::default()
+                },
+                ..Default::default()
+            }],
+            ..Default::default()
+        });
+
+        assert!(!should_auto_start_rebalance_after_init(
+            pool_meta_has_active_decommission(&pool_meta),
+            rebalance_meta
+                .as_ref()
+                .is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
         ));
     }
 

crates/ecstore/src/store/utils.rs

@@ -26,18 +26,18 @@ static STRICT_BUCKET_NAME_REGEX: LazyLock<Regex> =
 static NON_STRICT_BUCKET_NAME_REGEX: LazyLock<Regex> =
     LazyLock::new(|| Regex::new(r"^[A-Za-z0-9][A-Za-z0-9\.\-_:]{1,61}[A-Za-z0-9]$").expect("valid non-strict bucket name regex"));
 
-pub fn clean_metadata<S: std::hash::BuildHasher>(metadata: &mut HashMap<String, String, S>) {
+pub fn clean_metadata(metadata: &mut HashMap<String, String>) {
     remove_standard_storage_class(metadata);
     clean_metadata_keys(metadata, &["md5Sum", "etag", "expires", AMZ_OBJECT_TAGGING, "last-modified"]);
 }
 
-pub fn remove_standard_storage_class<S: std::hash::BuildHasher>(metadata: &mut HashMap<String, String, S>) {
+pub fn remove_standard_storage_class(metadata: &mut HashMap<String, String>) {
     if metadata.get(AMZ_STORAGE_CLASS) == Some(&STANDARD.to_string()) {
         metadata.remove(AMZ_STORAGE_CLASS);
     }
 }
 
-pub fn clean_metadata_keys<S: std::hash::BuildHasher>(metadata: &mut HashMap<String, String, S>, key_names: &[&str]) {
+pub fn clean_metadata_keys(metadata: &mut HashMap<String, String>, key_names: &[&str]) {
     for key in key_names {
         metadata.remove(key.to_owned());
     }

crates/filemeta/Cargo.toml

@@ -51,9 +51,6 @@ s3s = { workspace = true, features = ["minio"] }
 regex.workspace = true
 arc-swap.workspace = true
 
-# High-performance hashing
-ahash = { workspace = true, features = ["serde"] }
-
 [dev-dependencies]
 criterion = { workspace = true, features = ["html_reports"] }
 tempfile = { workspace = true }

crates/filemeta/src/fileinfo.rs

@@ -22,7 +22,6 @@ use rustfs_utils::http::{
     contains_key_str, get_consistent_str, get_str, has_internal_suffix, insert_str, is_encryption_metadata_key,
     starts_with_ignore_ascii_case,
 };
-use ahash::AHashMap;
 use s3s::dto::{RestoreStatus, Timestamp};
 use s3s::header::X_AMZ_RESTORE;
 use serde::de::{self, MapAccess, SeqAccess, Visitor, value::MapAccessDeserializer};
@@ -68,7 +67,7 @@ pub struct ObjectPartInfo {
     // Index holds the index of the part in the erasure coding
     pub index: Option<Bytes>,
     // Checksums holds checksums of the part
-    pub checksums: Option<AHashMap<String, String>>,
+    pub checksums: Option<HashMap<String, String>>,
     pub error: Option<String>,
 }
 
@@ -269,7 +268,7 @@ pub struct FileInfo {
     pub mode: Option<u32>,
     // WrittenByVersion is the unix time stamp of the version that created this version of the object
     pub written_by_version: Option<u64>,
-    pub metadata: AHashMap<String, String>,
+    pub metadata: HashMap<String, String>,
     pub parts: Vec<ObjectPartInfo>,
     pub erasure: ErasureInfo,
     // MarkDeleted marks this version as deleted
@@ -302,7 +301,7 @@ fn is_sensitive_metadata_key(key: &str) -> bool {
             .any(|prefix| starts_with_ignore_ascii_case(key, prefix))
 }
 
-struct RedactedMetadata<'a>(&'a AHashMap<String, String>);
+struct RedactedMetadata<'a>(&'a HashMap<String, String>);
 
 impl std::fmt::Debug for RedactedMetadata<'_> {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
@@ -426,7 +425,7 @@ struct FileInfoMapDef {
     size: i64,
     mode: Option<u32>,
     written_by_version: Option<u64>,
-    metadata: AHashMap<String, String>,
+    metadata: HashMap<String, String>,
     parts: Vec<ObjectPartInfo>,
     erasure: ErasureInfo,
     mark_deleted: bool,
@@ -1080,7 +1079,7 @@ impl FileInfo {
         mod_time: Option<OffsetDateTime>,
         actual_size: i64,
         index: Option<Bytes>,
-        checksums: Option<AHashMap<String, String>>,
+        checksums: Option<HashMap<String, String>>,
     ) {
         let part = ObjectPartInfo {
             etag,
@@ -1458,7 +1457,7 @@ pub fn parse_restore_obj_status(restore_hdr: &str) -> Result<RestoreStatus> {
     Err(Error::other(ERR_RESTORE_HDR_MALFORMED))
 }
 
-pub fn is_restored_object_on_disk<S: std::hash::BuildHasher>(meta: &HashMap<String, String, S>) -> bool {
+pub fn is_restored_object_on_disk(meta: &HashMap<String, String>) -> bool {
     if let Some(restore_hdr) = meta.get(X_AMZ_RESTORE.as_str())
         && let Ok(restore_status) = parse_restore_obj_status(restore_hdr)
     {
@@ -2136,7 +2135,7 @@ mod tests {
             -1_000_000i64..=1_000_000i64,
             optional_timestamp_strategy(),
             proptest::option::of(bytes_strategy(16)),
-            proptest::option::of(hash_map(small_string_strategy(), small_string_strategy(), 0..=3).prop_map(|m| m.into_iter().collect::<AHashMap<String, String>>())),
+            proptest::option::of(hash_map(small_string_strategy(), small_string_strategy(), 0..=3)),
             proptest::option::of(small_string_strategy()),
         )
             .prop_map(|(etag, number, size, actual_size, mod_time, index, checksums, error)| ObjectPartInfo {
@@ -2171,7 +2170,7 @@ mod tests {
                 -1_000_000i64..=1_000_000i64,
                 proptest::option::of(any::<u32>()),
                 proptest::option::of(any::<u64>()),
-                hash_map(small_string_strategy(), small_string_strategy(), 0..=4).prop_map(|m| m.into_iter().collect::<AHashMap<String, String>>()),
+                hash_map(small_string_strategy(), small_string_strategy(), 0..=4),
                 vec(object_part_info_strategy(), 0..=3),
                 erasure_info_strategy(),
                 any::<bool>(),

crates/filemeta/src/filemeta.rs

@@ -174,10 +174,10 @@ fn valid_target_delete_marker_version(arn: &str, version_id: &str) -> bool {
 /// included in the quorum hash, so such a divergence does surface — but as a
 /// quorum failure on an otherwise healthy object, which is not a state worth
 /// reaching. Merge the RPC metadata carrier instead, and only ever insert.
-fn persist_target_delete_marker_versions<S: std::hash::BuildHasher>(
+fn persist_target_delete_marker_versions(
     meta_sys: &mut HashMap<String, Vec<u8>>,
     versions: &HashMap<String, String>,
-    transport_metadata: &HashMap<String, String, S>,
+    transport_metadata: &HashMap<String, String>,
 ) {
     let mut bounded = BTreeMap::new();
     // A corrupt carrier means the dual internal prefixes disagreed. Do not merge

crates/filemeta/src/filemeta/inline_data.rs

@@ -181,7 +181,7 @@ mod tests {
             data_dir: Some(data_dir),
             size: 64 * 1024,
             mod_time: Some(OffsetDateTime::now_utc()),
-            metadata: metadata.into_iter().collect(),
+            metadata,
             erasure: ErasureInfo {
                 algorithm: ErasureAlgo::ReedSolomon.to_string(),
                 data_blocks: 4,

crates/filemeta/src/filemeta/version.rs

@@ -26,7 +26,7 @@ use super::msgp_decode::{
     PrependByteReader, prealloc_hint, read_exact_vec, read_nil_or_array_len, read_nil_or_map_len, skip_msgp_value,
 };
 use super::*;
-use crate::{AHashMap, ChecksumInfo, TransitionVersionState};
+use crate::{ChecksumInfo, TransitionVersionState};
 use rustfs_utils::HashAlgorithm;
 use rustfs_utils::http::{
     RUSTFS_INTERNAL_PREFIX, SUFFIX_CRC, SUFFIX_FREE_VERSION, SUFFIX_INLINE_DATA, SUFFIX_PART_CHECKSUMS, SUFFIX_PURGESTATUS,
@@ -377,7 +377,7 @@ impl<'a> DerivedInternalMetadata<'a> {
     }
 }
 
-struct UniquePartChecksums(AHashMap<String, String>);
+struct UniquePartChecksums(HashMap<String, String>);
 
 impl<'de> serde::Deserialize<'de> for UniquePartChecksums {
     fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
@@ -397,7 +397,7 @@ impl<'de> serde::Deserialize<'de> for UniquePartChecksums {
             where
                 A: serde::de::SeqAccess<'de>,
             {
-                let mut checksums = AHashMap::with_capacity(seq.size_hint().unwrap_or_default());
+                let mut checksums = HashMap::with_capacity(seq.size_hint().unwrap_or_default());
                 while let Some((key, value)) = seq.next_element::<(String, String)>()? {
                     if checksums.insert(key, value).is_some() {
                         return Err(serde::de::Error::custom("duplicate part checksum name"));
@@ -1477,7 +1477,7 @@ pub struct MetaObjectV1 {
     #[serde(rename = "Erasure")]
     pub erasure: MetaObjectV1Erasure,
     #[serde(rename = "Meta")]
-    pub meta: AHashMap<String, String>,
+    pub meta: HashMap<String, String>,
     #[serde(rename = "Parts")]
     pub parts: Vec<MetaObjectV1Part>,
     #[serde(rename = "VersionID")]
@@ -1543,7 +1543,7 @@ pub struct MetaObjectV1Part {
     #[serde(rename = "i")]
     pub index: Option<Bytes>,
     #[serde(rename = "crc")]
-    pub checksums: Option<AHashMap<String, String>>,
+    pub checksums: Option<HashMap<String, String>>,
     #[serde(rename = "err")]
     pub error: Option<String>,
 }
@@ -1887,7 +1887,7 @@ impl MetaObjectV1Part {
                 "i" => self.index = Some(Bytes::from(read_msgp_bin(rd)?)),
                 "crc" => {
                     let len = rmp::decode::read_map_len(rd)? as usize;
-                    let mut checksums = AHashMap::with_capacity(prealloc_hint(len));
+                    let mut checksums = HashMap::with_capacity(prealloc_hint(len));
                     for _ in 0..len {
                         checksums.insert(read_msgp_string(rd)?, read_msgp_string(rd)?);
                     }
@@ -2570,7 +2570,7 @@ impl MetaObject {
             Vec::new()
         };
 
-        let mut metadata = AHashMap::with_capacity(self.meta_user.len() + self.meta_sys.len());
+        let mut metadata = HashMap::with_capacity(self.meta_user.len() + self.meta_sys.len());
         for (k, v) in &self.meta_user {
             if k == AMZ_META_UNENCRYPTED_CONTENT_LENGTH || k == AMZ_META_UNENCRYPTED_CONTENT_MD5 {
                 continue;
@@ -2861,7 +2861,7 @@ impl From<FileInfo> for MetaObject {
     }
 }
 
-fn get_internal_replication_state<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> Option<ReplicationState> {
+fn get_internal_replication_state(metadata: &HashMap<String, String>) -> Option<ReplicationState> {
     let mut rs = ReplicationState::default();
     let mut has = false;
 
@@ -2942,7 +2942,7 @@ impl MetaDeleteMarker {
     }
 
     pub fn into_fileinfo(&self, volume: &str, path: &str, _all_parts: bool) -> Result<FileInfo> {
-        let metadata: AHashMap<String, String> = self
+        let metadata = self
             .meta_sys
             .clone()
             .into_iter()
@@ -5500,10 +5500,10 @@ mod tests {
     /// entirely, silently dropping the whole legacy body on re-marshal.
     #[test]
     fn legacy_version_body_round_trips_through_encode() {
-        let mut meta = AHashMap::new();
+        let mut meta = HashMap::new();
         meta.insert("content-type".to_string(), "application/octet-stream".to_string());
 
-        let mut crc = AHashMap::new();
+        let mut crc = HashMap::new();
         crc.insert("crc32c".to_string(), "deadbeef".to_string());
 
         let legacy = MetaObjectV1 {

crates/filemeta/src/lib.rs

@@ -22,12 +22,6 @@ mod replication;
 
 pub mod test_data;
 
-/// High-performance HashMap type alias using ahash instead of SipHash.
-pub type AHashMap<K, V> = ahash::AHashMap<K, V>;
-
-/// High-performance HashSet type alias using ahash.
-pub type AHashSet<K> = ahash::AHashSet<K>;
-
 pub use error::*;
 pub use fileinfo::*;
 pub use filemeta::*;

crates/filemeta/src/metacache.rs

@@ -14,8 +14,8 @@
 
 use crate::filemeta::msgp_decode::MAX_MSGP_ELEMENT_SIZE;
 use crate::{
-    AHashMap, Error, FileInfo, FileInfoOpts, FileInfoVersions, FileMeta, FileMetaShallowVersion, Result, VersionType,
-    get_file_info, merge_file_meta_versions, merge_file_meta_versions_with_write_quorum,
+    Error, FileInfo, FileInfoOpts, FileInfoVersions, FileMeta, FileMetaShallowVersion, Result, VersionType, get_file_info,
+    merge_file_meta_versions, merge_file_meta_versions_with_write_quorum,
 };
 use arc_swap::ArcSwapOption;
 use rmp::Marker;
@@ -1676,7 +1676,7 @@ mod tests {
     }
 
     fn metacache_entry_with_mod_time(mod_time: OffsetDateTime, etag: &str) -> MetaCacheEntry {
-        let mut metadata = AHashMap::new();
+        let mut metadata = HashMap::new();
         metadata.insert("etag".to_string(), etag.to_string());
 
         let mut meta = FileMeta::new();
@@ -1705,7 +1705,7 @@ mod tests {
         data_blocks: usize,
         parity_blocks: usize,
     ) -> MetaCacheEntry {
-        let mut metadata = AHashMap::new();
+        let mut metadata = HashMap::new();
         metadata.insert("etag".to_string(), etag.to_string());
 
         let mut fi = FileInfo::new("object", data_blocks, parity_blocks);
@@ -1730,7 +1730,7 @@ mod tests {
     fn metacache_entry_with_erasure_versions(versions: &[(OffsetDateTime, &str, usize, usize)]) -> MetaCacheEntry {
         let mut meta = FileMeta::new();
         for (idx, (mod_time, etag, data_blocks, parity_blocks)) in versions.iter().enumerate() {
-            let mut metadata = AHashMap::new();
+            let mut metadata = HashMap::new();
             metadata.insert("etag".to_string(), (*etag).to_string());
 
             let mut fi = FileInfo::new("object", *data_blocks, *parity_blocks);
@@ -1776,7 +1776,7 @@ mod tests {
     /// Build an entry holding a single object version with an explicit version id
     /// and mod_time, so a set of these can model DISJOINT per-disk version sets.
     fn metacache_entry_single_version(version_u128: u128, mod_time: OffsetDateTime, etag: &str) -> MetaCacheEntry {
-        let mut metadata = AHashMap::new();
+        let mut metadata = HashMap::new();
         metadata.insert("etag".to_string(), etag.to_string());
 
         let mut fi = FileInfo::new("object", 4, 2);

crates/storage-api/src/error.rs

@@ -81,6 +81,7 @@ pub enum StorageErrorCode {
     InsufficientWriteQuorum,
     PreconditionFailed,
     EntityTooSmall,
+    EntityTooLarge,
     InvalidRangeSpec,
     NotModified,
     InvalidPartNumber,
@@ -169,6 +170,7 @@ impl StorageErrorCode {
             Self::InsufficientWriteQuorum => 0x3A,
             Self::PreconditionFailed => 0x3B,
             Self::EntityTooSmall => 0x3C,
+            Self::EntityTooLarge => 0x56,
             Self::InvalidRangeSpec => 0x3D,
             Self::NotModified => 0x3E,
             Self::InvalidPartNumber => 0x3F,
@@ -257,6 +259,7 @@ impl StorageErrorCode {
             0x3A => Some(Self::InsufficientWriteQuorum),
             0x3B => Some(Self::PreconditionFailed),
             0x3C => Some(Self::EntityTooSmall),
+            0x56 => Some(Self::EntityTooLarge),
             0x3D => Some(Self::InvalidRangeSpec),
             0x3E => Some(Self::NotModified),
             0x3F => Some(Self::InvalidPartNumber),
@@ -350,6 +353,7 @@ mod tests {
         (StorageErrorCode::InsufficientWriteQuorum, 0x3A),
         (StorageErrorCode::PreconditionFailed, 0x3B),
         (StorageErrorCode::EntityTooSmall, 0x3C),
+        (StorageErrorCode::EntityTooLarge, 0x56),
         (StorageErrorCode::InvalidRangeSpec, 0x3D),
         (StorageErrorCode::NotModified, 0x3E),
         (StorageErrorCode::InvalidPartNumber, 0x3F),

crates/utils/Cargo.toml

@@ -58,9 +58,6 @@ transform-stream = { workspace = true, optional = true }
 url = { workspace = true, optional = true }
 zstd = { workspace = true, optional = true }
 
-# High-performance hashing
-ahash = { workspace = true, optional = true }
-
 [dev-dependencies]
 criterion = { workspace = true, features = ["html_reports"] }
 tempfile = { workspace = true }

crates/utils/src/http/header_compat.rs

@@ -89,7 +89,7 @@ pub fn is_object_encryption_marker(key: &str) -> bool {
 }
 
 /// Reads the logical object size recorded by encryption metadata.
-pub fn get_object_encryption_original_size<S: std::hash::BuildHasher>(metadata: &std::collections::HashMap<String, String, S>) -> std::io::Result<Option<i64>> {
+pub fn get_object_encryption_original_size(metadata: &std::collections::HashMap<String, String>) -> std::io::Result<Option<i64>> {
     let actual_size = super::get_str(metadata, super::SUFFIX_ACTUAL_SIZE);
     let size = get_case_insensitive(metadata, RUSTFS_ENCRYPTION_ORIGINAL_SIZE)
         .or_else(|| get_case_insensitive(metadata, SSEC_ORIGINAL_SIZE))
@@ -103,7 +103,7 @@ pub fn get_object_encryption_original_size<S: std::hash::BuildHasher>(metadata:
         .map_err(|error| std::io::Error::other(format!("Failed to parse encryption original size: {error}")))
 }
 
-fn get_case_insensitive<'a, S: std::hash::BuildHasher>(metadata: &'a std::collections::HashMap<String, String, S>, key: &str) -> Option<&'a str> {
+fn get_case_insensitive<'a>(metadata: &'a std::collections::HashMap<String, String>, key: &str) -> Option<&'a str> {
     metadata.get(key).map(String::as_str).or_else(|| {
         metadata
             .iter()

crates/utils/src/http/metadata_compat.rs

@@ -44,6 +44,8 @@ pub const SUFFIX_COMPRESSION: &str = "compression";
 pub const SUFFIX_REPLICATION_PRESERVE_CIPHERTEXT: &str = "replication-preserve-ciphertext";
 pub const SUFFIX_COMPRESSION_SIZE: &str = "compression-size";
 pub const SUFFIX_ACTUAL_SIZE: &str = "actual-size";
+/// Maximum logical object size for a capability-bound multipart upload.
+pub const SUFFIX_MAX_TOTAL_OBJECT_SIZE: &str = "max-total-object-size";
 pub const SUFFIX_ACTUAL_OBJECT_SIZE: &str = "actual-object-size";
 /// Used by replication; key stored with capital A
 pub const SUFFIX_ACTUAL_OBJECT_SIZE_CAP: &str = "Actual-Object-Size";
@@ -182,13 +184,13 @@ pub fn internal_key_rustfs(suffix: &str) -> String {
 
 // === String type (FileInfo.metadata, user_defined) ===
 
-pub fn insert_str<S: std::hash::BuildHasher>(map: &mut HashMap<String, String, S>, suffix: &str, value: String) {
+pub fn insert_str(map: &mut HashMap<String, String>, suffix: &str, value: String) {
     let (k1, k2) = both_keys(suffix);
     map.insert(k1, value.clone());
     map.insert(k2, value);
 }
 
-pub fn get_str<S: std::hash::BuildHasher>(map: &HashMap<String, String, S>, suffix: &str) -> Option<String> {
+pub fn get_str(map: &HashMap<String, String>, suffix: &str) -> Option<String> {
     if let Some(v) = with_internal_key(RUSTFS_INTERNAL_PREFIX, suffix, |k1| map.get(k1).cloned()) {
         return Some(v);
     }
@@ -202,7 +204,7 @@ pub fn get_str<S: std::hash::BuildHasher>(map: &HashMap<String, String, S>, suff
         .map(|(_, value)| value.clone())
 }
 
-fn get_consistent_value<'a, V: AsRef<[u8]>, S: std::hash::BuildHasher>(map: &'a HashMap<String, V, S>, suffix: &str) -> Option<&'a V> {
+fn get_consistent_value<'a, V: AsRef<[u8]>>(map: &'a HashMap<String, V>, suffix: &str) -> Option<&'a V> {
     let (rustfs_key, minio_key) = both_keys(suffix);
     let mut value = None;
     for (key, candidate) in map {
@@ -220,11 +222,11 @@ fn get_consistent_value<'a, V: AsRef<[u8]>, S: std::hash::BuildHasher>(map: &'a
 /// Returns a non-empty value when every compatibility key present for `suffix` agrees.
 /// A single RustFS or MinIO key is accepted for backward compatibility; conflicting or empty
 /// values return `None` so callers at destructive boundaries can fail closed.
-pub fn get_consistent_str<'a, S: std::hash::BuildHasher>(map: &'a HashMap<String, String, S>, suffix: &str) -> Option<&'a str> {
+pub fn get_consistent_str<'a>(map: &'a HashMap<String, String>, suffix: &str) -> Option<&'a str> {
     get_consistent_value(map, suffix).map(String::as_str)
 }
 
-pub fn contains_key_str<S: std::hash::BuildHasher>(map: &HashMap<String, String, S>, suffix: &str) -> bool {
+pub fn contains_key_str(map: &HashMap<String, String>, suffix: &str) -> bool {
     if with_internal_key(RUSTFS_INTERNAL_PREFIX, suffix, |k1| map.contains_key(k1)) {
         return true;
     }
@@ -236,7 +238,7 @@ pub fn contains_key_str<S: std::hash::BuildHasher>(map: &HashMap<String, String,
         .any(|key| key.eq_ignore_ascii_case(&k1) || key.eq_ignore_ascii_case(&k2))
 }
 
-pub fn remove_str<S: std::hash::BuildHasher>(map: &mut HashMap<String, String, S>, suffix: &str) {
+pub fn remove_str(map: &mut HashMap<String, String>, suffix: &str) {
     with_internal_key(RUSTFS_INTERNAL_PREFIX, suffix, |k1| map.remove(k1));
     with_internal_key(MINIO_INTERNAL_PREFIX, suffix, |k2| map.remove(k2));
     let (k1, k2) = both_keys(suffix);
@@ -285,7 +287,7 @@ pub fn strip_internal_prefix_preserving_case(key: &str) -> Option<&str> {
 
 /// Reads the bounded per-target delete-marker version map in one metadata scan.
 /// The boolean is set when matching metadata is malformed or compatibility keys disagree.
-pub fn target_delete_marker_versions<S: std::hash::BuildHasher>(map: &HashMap<String, String, S>) -> (HashMap<String, String>, bool) {
+pub fn target_delete_marker_versions(map: &HashMap<String, String>) -> (HashMap<String, String>, bool) {
     const MAX_ENTRIES: usize = 1_000;
     const MAX_ARN_LEN: usize = 1_024;
     const MAX_VERSION_ID_LEN: usize = 1_024;

docs/architecture/s3-tables-support-matrix.md

@@ -43,7 +43,7 @@ catalog extension.
 | PyIceberg | Automated | Creates namespace and table, appends rows, reloads, scans, probes metadata-location, refs, views, maintenance, diagnostics, and optional catalog-vended table credentials with an exact-prefix data-plane scope check. |
 | Spark Iceberg REST catalog | Manual/live harness | RustFS can generate pinned Spark/Iceberg package inputs, REST catalog properties, SQL, run commands, expected `row_count=2`, and a CI opt-in gate for namespace creation, table creation, append, refresh, count, and cleanup. Live Spark execution and commit-conflict probing are still manual validation items unless explicitly enabled in the runner. |
 | Trino Iceberg REST catalog | Manual/live harness | RustFS can generate catalog properties and a read-only `SELECT COUNT(*)` command for a table created by PyIceberg or Spark. Write compatibility is not claimed. |
-| DuckDB Iceberg | Manual/live harness | RustFS can generate `httpfs` and `iceberg` SQL using an operator-supplied current metadata location. Write and commit compatibility are not claimed. |
+| DuckDB Iceberg 1.5.5 | Automated | `duckdb_smoke.py` verifies the metadata-location read path and generic REST Catalog single-table create, insert, update, delete, merge, schema evolution, snapshots, concurrent writers, normal drop, PyIceberg cross-read, `/iceberg` with `s3` signing, and `/_iceberg` with `s3tables` signing. Staged create, purge-on-drop, and format v3 are verified as fail-closed boundaries. DuckDB's endpoint-disabled two-table mode is exercised without claiming cross-table atomicity. AWS `ENDPOINT_TYPE S3_TABLES` and catalog-vended credential integration are not claimed. |
 | StarRocks Iceberg REST catalog | Documented, not automated | External catalog read-path reference only. Write compatibility is not claimed. |
 | Databend | Manual/live harness | RustFS can generate an S3 stage read probe for table data files. RustFS does not claim Databend Iceberg REST Catalog integration yet. |
 | Snowflake Open Catalog / Iceberg integrations | Generated harness | RustFS can generate an operator-adapted external volume/catalog SQL template. Live RustFS interoperability is not claimed. |
@@ -52,10 +52,10 @@ catalog extension.
 
 | Area | Status | Current RustFS claim |
 |---|---|---|
-| Live conformance evidence | Manual/live harness | `engine_compatibility.py --print-live-evidence-schema` defines the required evidence schema and claim promotion boundaries. `pyiceberg_smoke.py --live-evidence-output` writes a validated PyIceberg evidence record after a successful live smoke run. |
+| Live conformance evidence | Automated for PyIceberg and DuckDB | `engine_compatibility.py --print-live-evidence-schema` defines the required evidence schema and claim promotion boundaries. `pyiceberg_smoke.py --live-evidence-output` and `duckdb_smoke.py --live-evidence-output` write validated client evidence records after successful live smoke runs. |
 | Production operations guide | Generated harness | `engine_compatibility.py --print-operations-guide` records command, evidence, pass criteria, and fail-closed signals for live conformance, durable backing cutover, maintenance, recovery, permissions, credential vending, and unsupported-claim governance. |
 | Vendor compatibility gap audit | Generated harness | `engine_compatibility.py --print-vendor-audit` records provider source URLs, catalog path and warehouse shapes, signing/auth models, error/permission/maintenance validation categories, and not-claimed boundaries for AWS S3 Tables, MinIO AIStor Tables, Cloudflare R2 Data Catalog, and Alibaba OSS Tables. |
-| Client claim promotion | Documented, not automated | PyIceberg remains the automated claim. Spark can be promoted only with recorded manual/live evidence; Trino and DuckDB read probes do not promote write compatibility; Snowflake and vendor profiles remain reference-only without repeatable live evidence. |
+| Client claim promotion | Automated for scoped clients | PyIceberg and DuckDB claims remain bounded by their repeatable smoke entrypoints and recorded versions. Spark can be promoted only with recorded manual/live evidence; Trino remains read-only; Snowflake and vendor profiles remain reference-only without repeatable live evidence. |
 
 ## Catalog API Matrix
 
@@ -242,6 +242,7 @@ compatibility claims:
 ```bash
 python3 scripts/table-catalog/test_pyiceberg_smoke.py
 python3 scripts/table-catalog/test_engine_compatibility.py
+python3 scripts/table-catalog/test_duckdb_smoke.py
 python3 scripts/table-catalog/test_failure_coverage.py
 python3 scripts/table-catalog/pyiceberg_smoke.py --print-client-matrix
 python3 scripts/table-catalog/pyiceberg_smoke.py --print-engine-compatibility
@@ -250,6 +251,7 @@ python3 scripts/table-catalog/pyiceberg_smoke.py --print-vendor-profiles
 python3 scripts/table-catalog/pyiceberg_smoke.py --print-production-readiness
 python3 scripts/table-catalog/engine_compatibility.py --print-vendor-audit
 python3 scripts/table-catalog/engine_compatibility.py --print-spark-config
+python3 scripts/table-catalog/engine_compatibility.py --print-duckdb-rest-sql
 python3 scripts/table-catalog/engine_compatibility.py \
   --profile aws-s3tables \
   --region us-east-1 \
@@ -304,9 +306,9 @@ Use conservative release wording that matches the matrix.
 Acceptable wording:
 
 > RustFS includes a core Iceberg REST Catalog-based S3 Tables implementation
-> with PyIceberg smoke coverage, table-aware S3 data-plane policy checks,
+> with PyIceberg and DuckDB smoke coverage, table-aware S3 data-plane policy checks,
 > controlled maintenance, catalog recovery diagnostics, manual conformance
-> input for Spark, Trino, DuckDB, Databend, and Snowflake, production-failure
+> input for Spark, Trino, Databend, and Snowflake, production-failure
 > probe harnesses, disaster-recovery and scale/fault rehearsal probes, and a
 > machine-readable production operations evidence guide.
 

docs/operations/kms-per-key-authorization.md

@@ -81,7 +81,7 @@ Scope and exemptions:
 
 - **SSE-KMS only.** SSE-S3 wraps its data key with a server-owned key the caller never names, and SSE-C never reaches KMS; both are exempt, matching AWS.
 - **The resolved key**, not the header. A bucket default encryption rule naming a KMS key is authorized the same way an explicit `x-amz-server-side-encryption-aws-kms-key-id` header is.
-- **Anonymous requests are exempt.** They have no identity policy to evaluate, and denying them would break public buckets holding SSE-KMS objects. They remain governed by bucket policy.
+- **Anonymous requests are denied.** An anonymous caller has no identity policy and therefore holds no `kms` grants, so under enforcement every anonymous read or write of an SSE-KMS object fails with `AccessDenied` — even when a bucket policy makes the bucket public. This matches AWS, where anonymous requests cannot use SSE-KMS objects at all, and it keeps the per-key gate meaningful: were anonymous requests exempt, any denied identity could bypass the gate on a public bucket by simply dropping its credentials. **A public bucket serving SSE-KMS objects is incompatible with enforcement** — serve public content unencrypted or under SSE-S3 instead. With enforcement off (the default), anonymous access to SSE-KMS objects remains governed by bucket policy alone. The server warns once per process when it first denies an anonymous request; per-request denials appear on audit entries (`kmsOutcome=failure`, `kmsErrorClass=access_denied`, empty requester identity) and at debug level.
 - **Internal work is exempt.** Replication, lifecycle transitions, healing and the scanner run as the system principal.
 - **Authorization runs before key state is checked**, so a denial cannot be used to probe whether a key exists, is disabled, or is pending deletion. The response is always `AccessDenied`.
 - **Multipart uploads are authorized at create time**, where the session data key is generated. Part uploads and completion reuse that envelope and are not re-authorized against the destination key.
@@ -89,13 +89,13 @@ Scope and exemptions:
 
 ## Migration
 
-Data-path enforcement is **off by default in this release** because it changes the outcome of requests that succeed today: an identity holding only `s3:PutObject` can currently encrypt under any key. Turning it on without preparing policies will produce `AccessDenied` on working workloads.
+Data-path enforcement is **off by default** because it changes the outcome of requests that succeed today: an identity holding only `s3:PutObject` can currently encrypt under any key. Turning it on without preparing policies will produce `AccessDenied` on working workloads.
 
 ```bash
 RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY=true
 ```
 
-The server logs the configured mode once at startup, and warns while enforcement is off. A later release defaults it to enabled.
+The server logs the configured mode once at startup, and warns while enforcement is off. Enforcement stays opt-in: there is no roadmap to flip the default.
 
 Recommended sequence:
 

docs/operations/presigned-multipart-size-limit.md

@@ -0,0 +1,55 @@
+# Presigned multipart total-size limit
+
+RustFS V2 supports an optional capability on a signed or SigV4-presigned
+`CreateMultipartUpload` request:
+
+```text
+x-rustfs-max-total-object-size=<unsigned 64-bit integer>
+```
+
+The backend must include the parameter before calculating the SigV4
+signature. It is part of the canonical query and cannot be added, removed, or
+changed by the browser. RustFS stores the verified limit in the multipart
+upload session and applies it to every `UploadPart` and to
+`CompleteMultipartUpload`.
+
+Backend pseudocode (the custom query must be present before signing):
+
+```text
+uri = "/photos/archive.zip?uploads"
+uri += "&x-rustfs-max-total-object-size=104857600"
+presigned_url = sigv4_presign("POST", uri, credentials)
+# Return presigned_url to the browser. Never append the parameter afterwards.
+```
+
+The resulting flow is:
+
+1. The backend signs `CreateMultipartUpload?...&x-rustfs-max-total-object-size=104857600`.
+2. RustFS verifies the SigV4 request and persists the limit with the upload ID.
+3. The browser uploads parts using the returned upload ID.
+4. RustFS rejects a part whose declared logical size would exceed the remaining
+   budget and rejects completion if the server-side part metadata exceeds the
+   limit.
+
+The limit is measured in logical object bytes (`actual_size`), not erasure,
+encryption, or compression bytes. Replacing an existing part uses replacement
+semantics: the old part size is removed before the new part size is admitted.
+Unknown-length parts are rejected for capped sessions rather than buffered
+without a bound. Capped parts are admitted under an upload-wide write lock
+before temporary shards are created and use a per-upload staging permit to
+bound local in-flight data. The distributed lock is released while the body is
+read and reacquired for the final check/rename, so Complete and Abort are not
+blocked behind a slow upload. The normal request-body stall timeout releases
+the staging permit when a client stops sending.
+
+The parameter is accepted only on `CreateMultipartUpload`. Supplying it on
+`UploadPart`, `CompleteMultipartUpload`, `AbortMultipartUpload`, listing, or
+copy operations returns `InvalidRequest`; those requests use the persisted
+session state. A multipart upload created without this parameter remains
+unlimited for backward compatibility. The V1 single-request capability
+(`x-rustfs-max-content-length`) is independent and is not a multipart limit.
+
+Because enforcement happens in the multipart data plane, every node that may
+receive requests for a capped upload must run the V2 implementation. During a
+rolling upgrade, route capped uploads only to upgraded nodes; older nodes treat
+the internal metadata as unknown and cannot enforce the limit.

docs/testing/ci-gates.md

@@ -119,6 +119,11 @@ pull-request gate.
 Use an exact preview tag for end-to-end release rehearsal. Manual dispatches
 are backfill/debug paths and do not prove the automatic `workflow_run` chain.
 
+A preview Release is internal validation state, not a deliverable: after the
+final tag's release is published, `cleanup-preview-releases` deletes every
+`<target>-preview.<N>` Release for that target. The tags themselves are kept, so
+the validated commit stays traceable.
+
 ## Evidence requirements
 
 A green check is useful only when it proves the intended behavior ran:

rustfs/src/app/multipart_usecase.rs

@@ -67,6 +67,7 @@ use super::storage_api::multipart_usecase::sse::{
 use super::storage_api::multipart_usecase::{
     StorageObjectInfo as ObjectInfo, StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader,
 };
+use crate::app::object::{guard_put_object_body_read_timeout, put_object_body_read_timeout};
 use crate::app::object_data_cache::{
     ObjectDataCacheAdapter, invalidate_object_data_cache_after_complete_multipart_success,
     invalidate_object_data_cache_before_mutation,
@@ -78,7 +79,11 @@ use crate::app::object_usecase::{
 use crate::app::runtime_sources::{
     AppContext, current_app_context, current_object_data_cache_for_context, current_object_store_handle_for_context,
 };
-use crate::auth::{VerifiedPresignedRequest, reject_presigned_put_max_content_length_for_other_operation};
+use crate::auth::{
+    VerifiedPresignedRequest, VerifiedSigV4Request, parse_presigned_multipart_max_total_object_size,
+    reject_presigned_multipart_max_total_object_size_for_other_operation,
+    reject_presigned_put_max_content_length_for_other_operation,
+};
 use crate::capacity::record_capacity_write;
 use crate::error::ApiError;
 use crate::table_catalog;
@@ -92,8 +97,9 @@ use rustfs_utils::CompressionAlgorithm;
 #[cfg(test)]
 use rustfs_utils::http::insert_header;
 use rustfs_utils::http::{
-    SUFFIX_REPLICATION_PRESERVE_CIPHERTEXT, SUFFIX_REPLICATION_STATUS, SUFFIX_REPLICATION_TIMESTAMP,
-    SUFFIX_SOURCE_REPLICATION_REQUEST, contains_key_str, get_header, get_source_scheme,
+    SUFFIX_MAX_TOTAL_OBJECT_SIZE, SUFFIX_REPLICATION_PRESERVE_CIPHERTEXT, SUFFIX_REPLICATION_STATUS,
+    SUFFIX_REPLICATION_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_REQUEST, contains_key_str, get_consistent_str, get_header,
+    get_source_scheme,
     headers::{AMZ_DECODED_CONTENT_LENGTH, AMZ_OBJECT_TAGGING, AMZ_STORAGE_CLASS},
     insert_str,
 };
@@ -108,6 +114,7 @@ use s3s::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
 use std::collections::{HashMap, HashSet};
 use std::str::FromStr;
 use std::sync::Arc;
+use std::time::Duration;
 use tokio::sync::RwLock;
 use tokio_util::io::StreamReader;
 use tracing::{instrument, warn};
@@ -226,6 +233,22 @@ fn create_multipart_upload_metadata(
     metadata
 }
 
+fn multipart_max_total_object_size(metadata: &HashMap<String, String>) -> S3Result<Option<u64>> {
+    if !contains_key_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE) {
+        return Ok(None);
+    }
+
+    let value = get_consistent_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE).ok_or_else(|| {
+        S3Error::with_message(
+            S3ErrorCode::InvalidRequest,
+            "multipart size capability metadata is missing or inconsistent".to_string(),
+        )
+    })?;
+    value.parse::<u64>().map(Some).map_err(|_| {
+        S3Error::with_message(S3ErrorCode::InvalidRequest, "multipart size capability metadata is invalid".to_string())
+    })
+}
+
 /// A multipart session advertises disk compression only when the staged-rollout
 /// switch (`RUSTFS_COMPRESSION_MULTIPART_ENABLED`) is on, the object key/headers
 /// qualify, AND the session is not an SSE-C ciphertext-passthrough replication
@@ -398,6 +421,11 @@ impl DefaultMultipartUsecase {
         &self,
         req: S3Request<AbortMultipartUploadInput>,
     ) -> S3Result<S3Response<AbortMultipartUploadOutput>> {
+        reject_presigned_multipart_max_total_object_size_for_other_operation(
+            &req.headers,
+            req.uri.query(),
+            req.extensions.get::<VerifiedSigV4Request>().is_some(),
+        )?;
         reject_presigned_put_max_content_length_for_other_operation(
             &req.headers,
             req.uri.query(),
@@ -444,6 +472,11 @@ impl DefaultMultipartUsecase {
         &self,
         req: S3Request<CompleteMultipartUploadInput>,
     ) -> S3Result<S3Response<CompleteMultipartUploadOutput>> {
+        reject_presigned_multipart_max_total_object_size_for_other_operation(
+            &req.headers,
+            req.uri.query(),
+            req.extensions.get::<VerifiedSigV4Request>().is_some(),
+        )?;
         reject_presigned_put_max_content_length_for_other_operation(
             &req.headers,
             req.uri.query(),
@@ -752,6 +785,11 @@ impl DefaultMultipartUsecase {
         &self,
         req: S3Request<CreateMultipartUploadInput>,
     ) -> S3Result<S3Response<CreateMultipartUploadOutput>> {
+        let multipart_max_total_object_size = parse_presigned_multipart_max_total_object_size(
+            &req.headers,
+            req.uri.query(),
+            req.extensions.get::<VerifiedSigV4Request>().is_some(),
+        )?;
         reject_presigned_put_max_content_length_for_other_operation(
             &req.headers,
             req.uri.query(),
@@ -807,6 +845,9 @@ impl DefaultMultipartUsecase {
         )?;
 
         let mut metadata = create_multipart_upload_metadata(input_metadata, &req.headers, tagging, storage_class.as_ref());
+        if let Some(limit) = multipart_max_total_object_size {
+            insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, limit.to_string());
+        }
 
         let has_explicit_object_lock_retention = object_lock_mode.is_some()
             || object_lock_retain_until_date.is_some()
@@ -978,6 +1019,11 @@ impl DefaultMultipartUsecase {
     #[instrument(level = "debug", skip(self, req))]
     #[hotpath::measure(impl_type = "MultipartUsecase")]
     pub async fn execute_upload_part(&self, req: S3Request<UploadPartInput>) -> S3Result<S3Response<UploadPartOutput>> {
+        reject_presigned_multipart_max_total_object_size_for_other_operation(
+            &req.headers,
+            req.uri.query(),
+            req.extensions.get::<VerifiedSigV4Request>().is_some(),
+        )?;
         reject_presigned_put_max_content_length_for_other_operation(
             &req.headers,
             req.uri.query(),
@@ -1006,6 +1052,40 @@ impl DefaultMultipartUsecase {
 
         let mut size = resolve_upload_part_size(&req.headers, content_length)?;
         let mut body_stream = body.ok_or_else(|| s3_error!(IncompleteBody))?;
+        let Some(store) = self.object_store() else {
+            return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
+        };
+        let fi = store
+            .get_multipart_info(&bucket, &key, &upload_id, &opts)
+            .await
+            .map_err(ApiError::from)?;
+        let max_total_object_size = multipart_max_total_object_size(&fi.user_defined)?;
+        if max_total_object_size.is_some() && size.is_some_and(|size| size < 0) {
+            return Err(S3Error::new(S3ErrorCode::UnexpectedContent));
+        }
+        if max_total_object_size.is_some() && size.is_none() {
+            return Err(S3Error::new(S3ErrorCode::UnexpectedContent));
+        }
+        if let (Some(limit), Some(size)) = (max_total_object_size, size)
+            && u64::try_from(size).is_ok_and(|size| size > limit)
+        {
+            return Err(S3Error::new(S3ErrorCode::EntityTooLarge));
+        }
+        if max_total_object_size.is_some() {
+            let request_id = req
+                .extensions
+                .get::<super::storage_api::multipart_usecase::request_context::RequestContext>()
+                .map(|ctx| ctx.request_id.clone())
+                .unwrap_or_default();
+            body_stream = guard_put_object_body_read_timeout(
+                body_stream,
+                &bucket,
+                &key,
+                &request_id,
+                content_length,
+                put_object_body_read_timeout().max(Duration::from_secs(rustfs_config::DEFAULT_HTTP_REQUEST_BODY_READ_TIMEOUT)),
+            );
+        }
 
         if size.is_none() {
             let mut total = 0i64;
@@ -1026,16 +1106,6 @@ impl DefaultMultipartUsecase {
             body_stream = StreamingBlob::wrap(stream);
         }
 
-        // Get multipart info early to check if managed encryption will be applied
-        let Some(store) = self.object_store() else {
-            return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
-        };
-
-        let fi = store
-            .get_multipart_info(&bucket, &key, &upload_id, &opts)
-            .await
-            .map_err(ApiError::from)?;
-
         let mut size = size.ok_or_else(|| s3_error!(UnexpectedContent))?;
         let ingress_stage_start = rustfs_io_metrics::put_stage_metrics_enabled().then(std::time::Instant::now);
 
@@ -1250,6 +1320,11 @@ impl DefaultMultipartUsecase {
         &self,
         req: S3Request<ListMultipartUploadsInput>,
     ) -> S3Result<S3Response<ListMultipartUploadsOutput>> {
+        reject_presigned_multipart_max_total_object_size_for_other_operation(
+            &req.headers,
+            req.uri.query(),
+            req.extensions.get::<VerifiedSigV4Request>().is_some(),
+        )?;
         reject_presigned_put_max_content_length_for_other_operation(
             &req.headers,
             req.uri.query(),
@@ -1302,6 +1377,11 @@ impl DefaultMultipartUsecase {
     }
 
     pub async fn execute_list_parts(&self, req: S3Request<ListPartsInput>) -> S3Result<S3Response<ListPartsOutput>> {
+        reject_presigned_multipart_max_total_object_size_for_other_operation(
+            &req.headers,
+            req.uri.query(),
+            req.extensions.get::<VerifiedSigV4Request>().is_some(),
+        )?;
         reject_presigned_put_max_content_length_for_other_operation(
             &req.headers,
             req.uri.query(),
@@ -1338,6 +1418,11 @@ impl DefaultMultipartUsecase {
         &self,
         req: S3Request<UploadPartCopyInput>,
     ) -> S3Result<S3Response<UploadPartCopyOutput>> {
+        reject_presigned_multipart_max_total_object_size_for_other_operation(
+            &req.headers,
+            req.uri.query(),
+            req.extensions.get::<VerifiedSigV4Request>().is_some(),
+        )?;
         reject_presigned_put_max_content_length_for_other_operation(
             &req.headers,
             req.uri.query(),
@@ -1441,6 +1526,7 @@ impl DefaultMultipartUsecase {
             .get_multipart_info(&bucket, &key, &upload_id, &dst_opts)
             .await
             .map_err(ApiError::from)?;
+        let destination_size_limit = multipart_max_total_object_size(&mp_info.user_defined)?;
         EncryptionRequest {
             bucket: &bucket,
             key: &key,
@@ -1523,19 +1609,25 @@ impl DefaultMultipartUsecase {
             return Err(s3_error!(PreconditionFailed));
         }
 
+        let source_logical_size = match src_info.get_actual_size() {
+            Ok(size) if size >= 0 => size,
+            Ok(_) | Err(_) if destination_size_limit.is_some() => {
+                return Err(S3Error::new(S3ErrorCode::UnexpectedContent));
+            }
+            Ok(_) | Err(_) => src_info.size,
+        };
+
         let (_start_offset, length) = if let Some(ref range_spec) = rs {
             // Copy-source ranges are expressed over the logical plaintext object.
             // Encrypted (and compressed) objects have a larger or smaller physical
             // representation, so validating against `size` rejects valid later parts.
-            let validation_size = src_info.get_actual_size().unwrap_or(src_info.size);
-
-            validate_copy_source_range_not_exceeds(range_spec, validation_size)?;
+            validate_copy_source_range_not_exceeds(range_spec, source_logical_size)?;
 
             range_spec
-                .get_offset_length(validation_size)
+                .get_offset_length(source_logical_size)
                 .map_err(|e| S3Error::with_message(S3ErrorCode::InvalidRange, e.to_string()))?
         } else {
-            (0, src_info.size)
+            (0, source_logical_size)
         };
 
         let is_disk_compressed =
@@ -2137,6 +2229,16 @@ mod tests {
         assert_eq!(metadata.get(AMZ_OBJECT_TAGGING), Some(&"project=rustfs".to_string()));
     }
 
+    #[test]
+    fn multipart_max_total_object_size_reads_compatible_internal_metadata() {
+        let mut metadata = HashMap::new();
+        insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "104857600".to_string());
+        assert_eq!(multipart_max_total_object_size(&metadata).unwrap(), Some(104_857_600));
+
+        metadata.insert("x-minio-internal-max-total-object-size".to_string(), "1".to_string());
+        assert!(multipart_max_total_object_size(&metadata).is_err());
+    }
+
     #[tokio::test]
     async fn execute_complete_multipart_upload_rejects_missing_parts_payload() {
         let input = CompleteMultipartUploadInput::builder()

rustfs/src/app/object/copy.rs

@@ -17,6 +17,71 @@
 use super::*;
 
 use crate::auth::{VerifiedPresignedRequest, reject_presigned_put_max_content_length_for_other_operation};
+use crate::error::ServerSideSourceReadError;
+
+struct CopySourceReadStream<R> {
+    inner: R,
+    remaining: i64,
+}
+
+impl<R> CopySourceReadStream<R> {
+    fn new(inner: R, expected_size: i64) -> Self {
+        Self {
+            inner,
+            remaining: expected_size.max(0),
+        }
+    }
+}
+
+fn copy_source_read_stream<R>(inner: R, expected_size: i64) -> CopySourceReadStream<R> {
+    CopySourceReadStream::new(inner, expected_size)
+}
+
+fn copy_source_read_error(source: std::io::Error) -> std::io::Error {
+    let kind = source.kind();
+    std::io::Error::new(kind, ServerSideSourceReadError::new("CopyObject", source))
+}
+
+fn copy_source_incomplete_body_error(remaining: i64) -> std::io::Error {
+    copy_source_read_error(std::io::Error::new(
+        std::io::ErrorKind::UnexpectedEof,
+        rustfs_rio::IncompleteBody { remaining },
+    ))
+}
+
+impl<R> AsyncRead for CopySourceReadStream<R>
+where
+    R: AsyncRead + Unpin,
+{
+    fn poll_read(self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
+        let this = self.get_mut();
+        let before = buf.filled().len();
+        match Pin::new(&mut this.inner).poll_read(cx, buf) {
+            Poll::Pending => Poll::Pending,
+            Poll::Ready(Err(err)) => Poll::Ready(Err(copy_source_read_error(err))),
+            Poll::Ready(Ok(())) => {
+                let read = buf.filled().len() - before;
+                if read == 0 {
+                    if this.remaining > 0 {
+                        return Poll::Ready(Err(copy_source_incomplete_body_error(this.remaining)));
+                    }
+                    return Poll::Ready(Ok(()));
+                }
+
+                let read = match i64::try_from(read) {
+                    Ok(read) => read,
+                    Err(_) => {
+                        return Poll::Ready(Err(copy_source_read_error(std::io::Error::other(
+                            "copy source read count exceeds i64::MAX",
+                        ))));
+                    }
+                };
+                this.remaining = this.remaining.saturating_sub(read);
+                Poll::Ready(Ok(()))
+            }
+        }
+    }
+}
 
 fn copy_namespace_lock_error(bucket: &str, object: &str, mode: &'static str, err: rustfs_lock::LockError) -> StorageError {
     match err {
@@ -580,11 +645,13 @@ impl DefaultObjectUsecase {
         let mut write_plan = WritePlan::new();
         let mut reader = if should_compress {
             let algorithm = CompressionAlgorithm::default();
-            let hrd = HashReader::from_stream(gr.stream, length, actual_size, None, None, false).map_err(ApiError::from)?;
+            let hrd = HashReader::from_stream(copy_source_read_stream(gr.stream, length), length, actual_size, None, None, false)
+                .map_err(ApiError::from)?;
             write_plan = write_plan.with_compression(algorithm);
             hrd
         } else {
-            HashReader::from_stream(gr.stream, length, actual_size, None, None, false).map_err(ApiError::from)?
+            HashReader::from_stream(copy_source_read_stream(gr.stream, length), length, actual_size, None, None, false)
+                .map_err(ApiError::from)?
         };
 
         // Give the destination object a checksum so CopyObject returns it and a later checksum-mode
@@ -835,6 +902,7 @@ mod tests {
     use http::{HeaderValue, Method};
     use s3s::dto::{ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule};
     use std::sync::Arc;
+    use tokio::io::AsyncReadExt;
 
     // A malformed bucket-default algorithm reaches this resolution only through
     // corrupt or hand-edited bucket metadata (PutBucketEncryption validates the
@@ -876,6 +944,23 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn copy_source_read_stream_maps_short_eof_to_service_unavailable() {
+        let source = std::io::Cursor::new(b"abc".to_vec());
+        let mut reader = HashReader::from_stream(copy_source_read_stream(source, 4), 4, 4, None, None, false)
+            .expect("copy source hash reader should build");
+
+        let mut output = Vec::new();
+        let err = reader
+            .read_to_end(&mut output)
+            .await
+            .expect_err("short copy source must fail before destination write succeeds");
+        let api_error = ApiError::from(err);
+
+        assert_eq!(api_error.code, S3ErrorCode::ServiceUnavailable);
+        assert_ne!(api_error.code, S3ErrorCode::IncompleteBody);
+    }
+
     #[tokio::test]
     async fn execute_copy_object_rejects_self_copy_without_replace_directive() {
         let input = CopyObjectInput::builder()

rustfs/src/app/object/mod.rs

@@ -195,6 +195,7 @@ pub(crate) use self::delete::*;
 pub(crate) use self::extract::*;
 pub(crate) use self::get::*;
 use self::put::*;
+pub(crate) use self::put::{guard_put_object_body_read_timeout, put_object_body_read_timeout};
 pub(crate) use self::shared::*;
 #[cfg(test)]
 use self::test_support::*;

rustfs/src/app/object/put.rs

@@ -90,7 +90,7 @@ fn resolve_put_object_authoritative_size(headers: &HeaderMap, content_length: Op
 /// Returns `Duration::ZERO` when disabled (`RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0`),
 /// in which case [`guard_put_object_body_read_timeout`] passes the body through
 /// untouched.
-fn put_object_body_read_timeout() -> Duration {
+pub(crate) fn put_object_body_read_timeout() -> Duration {
     Duration::from_secs(rustfs_utils::get_env_u64(
         rustfs_config::ENV_HTTP_REQUEST_BODY_READ_TIMEOUT,
         rustfs_config::DEFAULT_HTTP_REQUEST_BODY_READ_TIMEOUT,
@@ -260,7 +260,7 @@ impl ByteStream for RequestBodyReadTimeout {
 /// Wrap an incoming request body with [`RequestBodyReadTimeout`] unless the
 /// feature is disabled (`timeout == 0`), in which case the body is returned
 /// untouched. `remaining_length` is preserved via [`StreamingBlob::new`].
-fn guard_put_object_body_read_timeout(
+pub(crate) fn guard_put_object_body_read_timeout(
     body: StreamingBlob,
     bucket: &str,
     key: &str,
@@ -962,7 +962,7 @@ impl DefaultObjectUsecase {
         rustfs_io_metrics::record_put_object_stage_duration_from("app_bucket_validate", bucket_validate_stage_start);
 
         let put_admission = match get_concurrency_manager()
-            .admit_put_object()
+            .admit_put_object(size)
             .await
             .map_err(|_| s3_error!(InternalError, "foreground write admission closed"))?
         {

rustfs/src/auth.rs

@@ -53,6 +53,7 @@ const EVENT_SESSION_TOKEN_EXTRACTION: &str = "session_token_extraction";
 
 /// RustFS-specific query capability for a single presigned PutObject request.
 pub(crate) const RUSTFS_MAX_CONTENT_LENGTH_QUERY: &str = "x-rustfs-max-content-length";
+pub(crate) const RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY: &str = "x-rustfs-max-total-object-size";
 
 /// Inserted by the S3 access boundary after the upstream verifier accepts a
 /// request as SigV4 presigned. Downstream capability parsing must require this
@@ -60,6 +61,9 @@ pub(crate) const RUSTFS_MAX_CONTENT_LENGTH_QUERY: &str = "x-rustfs-max-content-l
 #[derive(Debug, Clone, Copy)]
 pub(crate) struct VerifiedPresignedRequest;
 
+#[derive(Debug, Clone, Copy)]
+pub(crate) struct VerifiedSigV4Request;
+
 /// Performs constant-time string comparison to prevent timing attacks.
 ///
 /// This function should be used when comparing sensitive values like passwords,
@@ -1111,6 +1115,92 @@ pub(crate) fn reject_presigned_put_max_content_length_for_other_operation(
     Ok(())
 }
 
+/// Parse the V2 multipart total-size capability after SigV4 authentication.
+/// Header-authenticated CreateMultipartUpload requests are accepted because the
+/// custom query is covered by the SigV4 canonical request; later multipart
+/// operations read the immutable value from the upload session metadata.
+pub(crate) fn parse_presigned_multipart_max_total_object_size(
+    header: &HeaderMap,
+    query: Option<&str>,
+    verified_sigv4: bool,
+) -> S3Result<Option<u64>> {
+    let Some(query) = query else {
+        return Ok(None);
+    };
+
+    let mut value = None;
+    let mut decoded_query = Vec::new();
+    for (name, candidate) in form_urlencoded::parse(query.as_bytes()) {
+        decoded_query.push((name.to_string(), candidate.to_string()));
+        if name == RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY {
+            if value.is_some() {
+                return Err(S3Error::with_message(
+                    S3ErrorCode::InvalidRequest,
+                    format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} must appear exactly once"),
+                ));
+            }
+            value = Some(candidate.into_owned());
+        } else if name.eq_ignore_ascii_case(RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY) {
+            return Err(S3Error::with_message(
+                S3ErrorCode::InvalidRequest,
+                format!("query parameter name must be exactly {RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY}"),
+            ));
+        }
+    }
+
+    let Some(value) = value else {
+        return Ok(None);
+    };
+
+    let auth_type = get_request_auth_type_with_query(header, Some(query));
+    let is_presigned = matches!(auth_type, AuthType::Presigned);
+    let is_header_signed = matches!(auth_type, AuthType::Signed);
+    let complete_presigned_query = [
+        ("x-amz-algorithm", "AWS4-HMAC-SHA256"),
+        ("x-amz-date", ""),
+        ("x-amz-expires", ""),
+        ("x-amz-signedheaders", ""),
+        ("x-amz-credential", ""),
+        ("x-amz-signature", ""),
+    ]
+    .into_iter()
+    .all(|(name, expected)| {
+        decoded_query
+            .iter()
+            .find(|(candidate, _)| candidate.eq_ignore_ascii_case(name))
+            .is_some_and(|(_, candidate)| !candidate.is_empty() && (expected.is_empty() || candidate == expected))
+    });
+
+    let authenticated = verified_sigv4 && (is_header_signed || (is_presigned && complete_presigned_query));
+    if !authenticated {
+        return Err(S3Error::with_message(
+            S3ErrorCode::InvalidRequest,
+            format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} requires a verified SigV4 request"),
+        ));
+    }
+
+    value.parse::<u64>().map(Some).map_err(|_| {
+        S3Error::with_message(
+            S3ErrorCode::InvalidRequest,
+            format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} must be a non-negative 64-bit integer"),
+        )
+    })
+}
+
+pub(crate) fn reject_presigned_multipart_max_total_object_size_for_other_operation(
+    header: &HeaderMap,
+    query: Option<&str>,
+    verified_sigv4: bool,
+) -> S3Result<()> {
+    if parse_presigned_multipart_max_total_object_size(header, query, verified_sigv4)?.is_some() {
+        return Err(S3Error::with_message(
+            S3ErrorCode::InvalidRequest,
+            format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} is only supported for CreateMultipartUpload"),
+        ));
+    }
+    Ok(())
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -1816,6 +1906,52 @@ mod tests {
         }
     }
 
+    #[test]
+    fn multipart_max_total_object_size_requires_signed_create_request() {
+        let headers = HeaderMap::new();
+        let signed_prefix = "X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20260827T000000Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Credential=test/20260827/us-east-1/s3/aws4_request&X-Amz-Signature=signature";
+        let query = format!("{signed_prefix}&x-rustfs-max-total-object-size=104857600");
+
+        assert_eq!(
+            parse_presigned_multipart_max_total_object_size(&headers, Some(&query), true).unwrap(),
+            Some(104_857_600)
+        );
+        assert_eq!(
+            reject_presigned_multipart_max_total_object_size_for_other_operation(&headers, Some(&query), true)
+                .unwrap_err()
+                .code(),
+            &S3ErrorCode::InvalidRequest
+        );
+    }
+
+    #[test]
+    fn multipart_max_total_object_size_rejects_tampering_and_invalid_values() {
+        let headers = HeaderMap::new();
+        let signed_prefix = "X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20260827T000000Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Credential=test/20260827/us-east-1/s3/aws4_request&X-Amz-Signature=signature";
+        for query in [
+            "x-rustfs-max-total-object-size=1",
+            "X-RustFS-Max-Total-Object-Size=1",
+            "X-Amz-Algorithm=AWS4-HMAC-SHA256&x-rustfs-max-total-object-size=1&x-rustfs-max-total-object-size=2",
+            "X-Amz-Algorithm=AWS4-HMAC-SHA256&x-rustfs-max-total-object-size=-1",
+            "X-Amz-Algorithm=AWS4-HMAC-SHA256&x-rustfs-max-total-object-size=18446744073709551616",
+        ] {
+            assert_eq!(
+                parse_presigned_multipart_max_total_object_size(&headers, Some(query), true)
+                    .unwrap_err()
+                    .code(),
+                &S3ErrorCode::InvalidRequest
+            );
+        }
+
+        let forged = format!("{signed_prefix}&x-rustfs-max-total-object-size=1");
+        assert_eq!(
+            parse_presigned_multipart_max_total_object_size(&headers, Some(&forged), false)
+                .unwrap_err()
+                .code(),
+            &S3ErrorCode::InvalidRequest
+        );
+    }
+
     #[test]
     fn test_credentials_is_expired() {
         let mut cred = create_test_credentials();

rustfs/src/error.rs

@@ -34,6 +34,32 @@ impl std::fmt::Display for UploadLimitExceeded {
 
 impl std::error::Error for UploadLimitExceeded {}
 
+/// Marks a server-side object/source reader failure that must not be reported as
+/// a malformed client request body.
+#[derive(Debug)]
+pub(crate) struct ServerSideSourceReadError {
+    operation: &'static str,
+    source: std::io::Error,
+}
+
+impl ServerSideSourceReadError {
+    pub(crate) const fn new(operation: &'static str, source: std::io::Error) -> Self {
+        Self { operation, source }
+    }
+}
+
+impl std::fmt::Display for ServerSideSourceReadError {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{} source read failed: {}", self.operation, self.source)
+    }
+}
+
+impl std::error::Error for ServerSideSourceReadError {
+    fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
+        Some(&self.source)
+    }
+}
+
 #[derive(Debug)]
 pub struct ApiError {
     pub code: S3ErrorCode,
@@ -302,6 +328,17 @@ impl From<StorageError> for ApiError {
             };
         }
 
+        if let StorageError::Io(ref io_err) = err
+            && let Some(inner) = io_err.get_ref()
+            && error_chain_has_type::<ServerSideSourceReadError>(inner)
+        {
+            return ApiError {
+                code: S3ErrorCode::ServiceUnavailable,
+                message: ApiError::error_code_to_message(&S3ErrorCode::ServiceUnavailable),
+                source: Some(Box::new(err)),
+            };
+        }
+
         if let StorageError::Io(ref io_err) = err
             && io_err
                 .get_ref()
@@ -340,15 +377,15 @@ impl From<StorageError> for ApiError {
             StorageError::ObjectNameInvalid(_, _) => S3ErrorCode::InvalidArgument,
             StorageError::BucketExists(_) => S3ErrorCode::BucketAlreadyOwnedByYou,
             StorageError::StorageFull => S3ErrorCode::ServiceUnavailable,
-            StorageError::SlowDown
-            | StorageError::FaultyDisk
+            StorageError::SlowDown => S3ErrorCode::SlowDown,
+            StorageError::FaultyDisk
             | StorageError::FaultyRemoteDisk
             | StorageError::DiskNotFound
-            | StorageError::TooManyOpenFiles => S3ErrorCode::SlowDown,
+            | StorageError::TooManyOpenFiles => S3ErrorCode::ServiceUnavailable,
             StorageError::ErasureReadQuorum
             | StorageError::InsufficientReadQuorum(_, _)
             | StorageError::ErasureWriteQuorum
-            | StorageError::InsufficientWriteQuorum(_, _) => S3ErrorCode::SlowDown,
+            | StorageError::InsufficientWriteQuorum(_, _) => S3ErrorCode::ServiceUnavailable,
             StorageError::NamespaceLockQuorumUnavailable { .. } => S3ErrorCode::ServiceUnavailable,
             StorageError::QuotaExceeded { .. } => S3ErrorCode::InvalidRequest,
             StorageError::Lock(_) => S3ErrorCode::ServiceUnavailable,
@@ -372,6 +409,7 @@ impl From<StorageError> for ApiError {
             StorageError::ObjectExistsAsDirectory(_, _) => S3ErrorCode::InvalidArgument,
             StorageError::InvalidPart(_, _, _) => S3ErrorCode::InvalidPart,
             StorageError::EntityTooSmall(_, _, _) => S3ErrorCode::EntityTooSmall,
+            StorageError::EntityTooLarge(_, _) => S3ErrorCode::EntityTooLarge,
             StorageError::PreconditionFailed => S3ErrorCode::PreconditionFailed,
             StorageError::NotModified => S3ErrorCode::NotModified,
             StorageError::InvalidRangeSpec(_) => S3ErrorCode::InvalidRange,
@@ -434,6 +472,13 @@ impl From<std::io::Error> for ApiError {
                     source: Some(Box::new(err)),
                 };
             }
+            if error_chain_has_type::<ServerSideSourceReadError>(inner) {
+                return ApiError {
+                    code: S3ErrorCode::ServiceUnavailable,
+                    message: ApiError::error_code_to_message(&S3ErrorCode::ServiceUnavailable),
+                    source: Some(Box::new(err)),
+                };
+            }
             if error_chain_has_type::<rustfs_rio::IncompleteBody>(inner) {
                 return ApiError {
                     code: S3ErrorCode::IncompleteBody,
@@ -612,6 +657,22 @@ mod tests {
         }
     }
 
+    #[test]
+    fn server_side_source_read_error_maps_to_service_unavailable_before_incomplete_body() {
+        let short_source = IoError::new(ErrorKind::UnexpectedEof, rustfs_rio::IncompleteBody { remaining: 17 });
+        let marker = ServerSideSourceReadError::new("CopyObject", short_source);
+
+        let api_error = ApiError::from(IoError::new(ErrorKind::UnexpectedEof, marker));
+        assert_eq!(api_error.code, S3ErrorCode::ServiceUnavailable);
+        assert_ne!(api_error.code, S3ErrorCode::IncompleteBody);
+
+        let short_source = IoError::new(ErrorKind::UnexpectedEof, rustfs_rio::IncompleteBody { remaining: 17 });
+        let marker = ServerSideSourceReadError::new("CopyObject", short_source);
+        let api_error = ApiError::from(StorageError::Io(IoError::new(ErrorKind::UnexpectedEof, marker)));
+        assert_eq!(api_error.code, S3ErrorCode::ServiceUnavailable);
+        assert_ne!(api_error.code, S3ErrorCode::IncompleteBody);
+    }
+
     #[test]
     fn test_api_error_surfaces_invalid_argument_reason() {
         let err = StorageError::InvalidArgument(
@@ -784,14 +845,20 @@ mod tests {
             (StorageError::BucketExists("test".into()), S3ErrorCode::BucketAlreadyOwnedByYou),
             (StorageError::StorageFull, S3ErrorCode::ServiceUnavailable),
             (StorageError::SlowDown, S3ErrorCode::SlowDown),
-            (StorageError::FaultyDisk, S3ErrorCode::SlowDown),
-            (StorageError::FaultyRemoteDisk, S3ErrorCode::SlowDown),
-            (StorageError::DiskNotFound, S3ErrorCode::SlowDown),
-            (StorageError::TooManyOpenFiles, S3ErrorCode::SlowDown),
-            (StorageError::ErasureReadQuorum, S3ErrorCode::SlowDown),
-            (StorageError::InsufficientReadQuorum("test".into(), "test".into()), S3ErrorCode::SlowDown),
-            (StorageError::ErasureWriteQuorum, S3ErrorCode::SlowDown),
-            (StorageError::InsufficientWriteQuorum("test".into(), "test".into()), S3ErrorCode::SlowDown),
+            (StorageError::FaultyDisk, S3ErrorCode::ServiceUnavailable),
+            (StorageError::FaultyRemoteDisk, S3ErrorCode::ServiceUnavailable),
+            (StorageError::DiskNotFound, S3ErrorCode::ServiceUnavailable),
+            (StorageError::TooManyOpenFiles, S3ErrorCode::ServiceUnavailable),
+            (StorageError::ErasureReadQuorum, S3ErrorCode::ServiceUnavailable),
+            (
+                StorageError::InsufficientReadQuorum("test".into(), "test".into()),
+                S3ErrorCode::ServiceUnavailable,
+            ),
+            (StorageError::ErasureWriteQuorum, S3ErrorCode::ServiceUnavailable),
+            (
+                StorageError::InsufficientWriteQuorum("test".into(), "test".into()),
+                S3ErrorCode::ServiceUnavailable,
+            ),
             (
                 StorageError::NamespaceLockQuorumUnavailable {
                     mode: "write",

rustfs/src/storage/access.rs

@@ -16,9 +16,10 @@ use super::ObjectOptions;
 use super::ecfs::FS;
 use super::{ECStore, PolicySys, ReplicationStatusType, StorageError, get_lock_acquire_timeout, is_err_bucket_not_found};
 use crate::auth::{
-    AuthType, RUSTFS_MAX_CONTENT_LENGTH_QUERY, VerifiedPresignedRequest, check_key_valid_with_context,
-    get_condition_values_with_client_info, get_condition_values_with_query_and_client_info, get_request_auth_type_with_query,
-    get_session_token, parse_presigned_put_max_content_length,
+    AuthType, RUSTFS_MAX_CONTENT_LENGTH_QUERY, RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY, VerifiedPresignedRequest,
+    VerifiedSigV4Request, check_key_valid_with_context, get_condition_values_with_client_info,
+    get_condition_values_with_query_and_client_info, get_request_auth_type_with_query, get_session_token,
+    parse_presigned_multipart_max_total_object_size, parse_presigned_put_max_content_length,
 };
 use crate::error::ApiError;
 use crate::license::license_check;
@@ -1771,7 +1772,9 @@ impl S3Access for FS {
 
         // Publish this server's context slot so downstream data-plane handlers
         // resolve the same store (backlog#1052 S6).
-        let verified_presigned = matches!(get_request_auth_type_with_query(cx.headers(), cx.uri().query()), AuthType::Presigned);
+        let auth_type = get_request_auth_type_with_query(cx.headers(), cx.uri().query());
+        let verified_presigned = matches!(auth_type, AuthType::Presigned);
+        let verified_sigv4 = matches!(auth_type, AuthType::Presigned | AuthType::Signed);
         {
             let ext = cx.extensions_mut();
             ext.insert(self.server_ctx().clone());
@@ -1779,6 +1782,9 @@ impl S3Access for FS {
             if verified_presigned {
                 ext.insert(VerifiedPresignedRequest);
             }
+            if verified_sigv4 {
+                ext.insert(VerifiedSigV4Request);
+            }
         }
 
         // The size capability is intentionally scoped to the single-object
@@ -1793,6 +1799,14 @@ impl S3Access for FS {
                 format!("{RUSTFS_MAX_CONTENT_LENGTH_QUERY} is only supported for presigned PutObject"),
             ));
         }
+        if parse_presigned_multipart_max_total_object_size(cx.headers(), cx.uri().query(), verified_sigv4)?.is_some()
+            && cx.s3_op().name() != "CreateMultipartUpload"
+        {
+            return Err(S3Error::with_message(
+                S3ErrorCode::InvalidRequest,
+                format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} is only supported for CreateMultipartUpload"),
+            ));
+        }
         license_check().map_err(|er| match er.kind() {
             std::io::ErrorKind::PermissionDenied => s3_error!(AccessDenied, "{er}"),
             _ => {

rustfs/src/storage/concurrency/manager.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-//! Concurrency manager for coordinating concurrent GetObject requests.
+//! Concurrency manager for coordinating concurrent GetObject and PutObject requests.
 
 use super::io_schedule::{
     IoLoadLevel, IoLoadMetrics, IoPriority, IoPriorityQueue, IoPriorityQueueConfig, IoQueueStatus, IoSchedulerConfig, IoStrategy,
@@ -34,6 +34,8 @@ use std::time::Duration;
 use tokio::sync::Semaphore;
 use tracing::debug;
 
+const DERIVED_LARGE_PUT_ADMISSION_LIMIT_MAX: usize = 32;
+
 /// Global concurrency manager instance
 pub(crate) static CONCURRENCY_MANAGER: LazyLock<ConcurrencyManager> = LazyLock::new(ConcurrencyManager::new);
 
@@ -65,11 +67,8 @@ pub struct ConcurrencyManager {
     bandwidth_monitor: Arc<Mutex<BandwidthMonitor>>,
     /// Metrics collector for I/O latency tracking (P50, P95, P99)
     metrics_collector: Arc<MetricsCollector>,
-    /// Experimental fixed-count foreground PutObject admission gate.
-    put_admission_semaphore: Arc<Semaphore>,
-    put_admission_enabled: bool,
-    put_admission_limit: usize,
-    put_admission_wait_timeout: Duration,
+    /// Foreground PutObject admission policy, resolved once at startup.
+    put_admission_policy: PutAdmissionPolicy,
 }
 
 impl std::fmt::Debug for ConcurrencyManager {
@@ -127,10 +126,201 @@ pub enum PutObjectAdmission {
     /// Request is admitted and must hold the permit until the store write
     /// returns or the request fails before mutation.
     Admitted(tokio::sync::OwnedSemaphorePermit),
-    /// The fixed-count gate stayed full until the configured wait timeout.
+    /// The selected foreground PUT admission gate stayed full until the configured wait timeout.
     Rejected,
 }
 
+#[derive(Clone)]
+struct PutAdmissionGate {
+    semaphore: Arc<Semaphore>,
+    limit: usize,
+    wait_timeout: Duration,
+}
+
+impl PutAdmissionGate {
+    fn new(limit: usize, wait_timeout: Duration) -> Self {
+        Self {
+            semaphore: Arc::new(Semaphore::new(limit)),
+            limit,
+            wait_timeout,
+        }
+    }
+
+    fn active(&self) -> usize {
+        self.limit.saturating_sub(self.semaphore.available_permits())
+    }
+
+    async fn admit(&self) -> Result<PutObjectAdmission, tokio::sync::AcquireError> {
+        if self.wait_timeout.is_zero() {
+            return Ok(match self.semaphore.clone().try_acquire_owned() {
+                Ok(permit) => PutObjectAdmission::Admitted(permit),
+                Err(tokio::sync::TryAcquireError::NoPermits) => PutObjectAdmission::Rejected,
+                Err(tokio::sync::TryAcquireError::Closed) => PutObjectAdmission::Rejected,
+            });
+        }
+
+        match tokio::time::timeout(self.wait_timeout, self.semaphore.clone().acquire_owned()).await {
+            Ok(permit) => Ok(PutObjectAdmission::Admitted(permit?)),
+            Err(_) => Ok(PutObjectAdmission::Rejected),
+        }
+    }
+}
+
+#[derive(Clone)]
+enum PutAdmissionPolicy {
+    /// Strict admission was explicitly enabled with limit `0`.
+    Disabled,
+    /// No hard PUT gate is configured; foreground write snapshots use the
+    /// existing active request counter as a soft pressure signal.
+    LegacyCounterOnly,
+    /// Explicit all-PUT admission gate.
+    Strict(PutAdmissionGate),
+    /// Default large/unknown-size PUT admission gate.
+    Large { gate: PutAdmissionGate, min_size_bytes: usize },
+}
+
+impl PutAdmissionPolicy {
+    fn from_env(max_disk_reads: usize) -> Self {
+        let strict_enabled = rustfs_utils::get_env_bool(
+            rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_ENABLE,
+            rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE,
+        );
+        if strict_enabled {
+            let strict_limit = rustfs_utils::get_env_usize(
+                rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_LIMIT,
+                rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_LIMIT,
+            );
+            let strict_wait_timeout = Duration::from_millis(rustfs_utils::get_env_u64(
+                rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
+                rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
+            ));
+            return if strict_limit == 0 {
+                Self::Disabled
+            } else {
+                Self::Strict(PutAdmissionGate::new(strict_limit, strict_wait_timeout))
+            };
+        }
+
+        let large_enabled = rustfs_utils::get_env_bool(
+            rustfs_config::ENV_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE,
+            rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE,
+        );
+        if !large_enabled {
+            return Self::LegacyCounterOnly;
+        }
+
+        let large_limit = derive_large_put_admission_limit(
+            rustfs_utils::get_env_usize(
+                rustfs_config::ENV_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT,
+                rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT,
+            ),
+            max_disk_reads,
+        );
+        let min_size_bytes = rustfs_utils::get_env_usize(
+            rustfs_config::ENV_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES,
+            rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES,
+        );
+        let wait_timeout = Duration::from_millis(rustfs_utils::get_env_u64(
+            rustfs_config::ENV_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
+            rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
+        ));
+
+        Self::Large {
+            gate: PutAdmissionGate::new(large_limit, wait_timeout),
+            min_size_bytes,
+        }
+    }
+
+    #[cfg(test)]
+    fn strict_for_test(enabled: bool, limit: usize, wait_timeout: Duration) -> Self {
+        if enabled {
+            if limit == 0 {
+                Self::Disabled
+            } else {
+                Self::Strict(PutAdmissionGate::new(limit, wait_timeout))
+            }
+        } else {
+            Self::LegacyCounterOnly
+        }
+    }
+
+    #[cfg(test)]
+    fn large_for_test(enabled: bool, limit: usize, min_size_bytes: usize, wait_timeout: Duration) -> Self {
+        if enabled && limit > 0 {
+            Self::Large {
+                gate: PutAdmissionGate::new(limit, wait_timeout),
+                min_size_bytes,
+            }
+        } else {
+            Self::LegacyCounterOnly
+        }
+    }
+
+    async fn admit(&self, size: i64) -> Result<PutObjectAdmission, tokio::sync::AcquireError> {
+        match self {
+            Self::Disabled | Self::LegacyCounterOnly => Ok(PutObjectAdmission::Disabled),
+            Self::Strict(gate) => gate.admit().await,
+            Self::Large { gate, min_size_bytes } if should_gate_large_put(size, *min_size_bytes) => gate.admit().await,
+            Self::Large { .. } => Ok(PutObjectAdmission::Disabled),
+        }
+    }
+
+    fn snapshot(&self, legacy_limit: usize) -> WorkloadAdmissionSnapshot {
+        match self {
+            Self::Disabled => put_admission_snapshot(0, 0, None),
+            Self::LegacyCounterOnly => put_admission_snapshot(PutObjectGuard::concurrent_count(), legacy_limit, None),
+            Self::Strict(gate) => {
+                put_admission_snapshot(gate.active(), gate.limit, Some("foreground write admission permits exhausted"))
+            }
+            Self::Large { gate, .. } => {
+                put_admission_snapshot(gate.active(), gate.limit, Some("large foreground write admission permits exhausted"))
+            }
+        }
+    }
+}
+
+fn put_admission_snapshot(active: usize, limit: usize, hard_gate_reason: Option<&'static str>) -> WorkloadAdmissionSnapshot {
+    let state = if limit == 0 {
+        AdmissionState::Disabled
+    } else if active >= limit {
+        AdmissionState::Saturated
+    } else {
+        AdmissionState::Open
+    };
+
+    let admission =
+        WorkloadAdmissionSnapshot::new(WorkloadClass::ForegroundWrite, state).with_counts(Some(active), None, Some(limit));
+
+    match state {
+        AdmissionState::Disabled => admission.with_reason("foreground write admission disabled"),
+        AdmissionState::Saturated => {
+            admission.with_reason(hard_gate_reason.unwrap_or("foreground write concurrency reached local pressure limit"))
+        }
+        _ => admission,
+    }
+}
+
+fn derive_large_put_admission_limit(configured_limit: usize, max_disk_reads: usize) -> usize {
+    if configured_limit > 0 {
+        return configured_limit;
+    }
+
+    let scheduler_base = if max_disk_reads == 0 {
+        rustfs_config::DEFAULT_OBJECT_MAX_CONCURRENT_DISK_READS
+    } else {
+        max_disk_reads
+    };
+    scheduler_base.div_ceil(2).clamp(1, DERIVED_LARGE_PUT_ADMISSION_LIMIT_MAX)
+}
+
+fn should_gate_large_put(size: i64, min_size_bytes: usize) -> bool {
+    if min_size_bytes == 0 || size < 0 {
+        return true;
+    }
+
+    usize::try_from(size).is_ok_and(|size| size >= min_size_bytes)
+}
+
 impl ConcurrencyManager {
     /// Create a new concurrency manager with default settings
     ///
@@ -177,18 +367,7 @@ impl ConcurrencyManager {
         // Initialize metrics collector for I/O latency tracking
         // Keep 1000 samples for P95/P99 calculation
         let metrics_collector = Arc::new(MetricsCollector::new(performance_metrics, 1000));
-        let put_admission_enabled = rustfs_utils::get_env_bool(
-            rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_ENABLE,
-            rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE,
-        );
-        let put_admission_limit = rustfs_utils::get_env_usize(
-            rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_LIMIT,
-            rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_LIMIT,
-        );
-        let put_admission_wait_timeout = Duration::from_millis(rustfs_utils::get_env_u64(
-            rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
-            rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
-        ));
+        let put_admission_policy = PutAdmissionPolicy::from_env(max_disk_reads);
 
         // Build queue config directly from scheduler config.
         let queue_config = IoPriorityQueueConfig::from_scheduler_config(&scheduler_config);
@@ -204,10 +383,7 @@ impl ConcurrencyManager {
             pattern_detector,
             bandwidth_monitor,
             metrics_collector,
-            put_admission_semaphore: Arc::new(Semaphore::new(if put_admission_enabled { put_admission_limit } else { 0 })),
-            put_admission_enabled,
-            put_admission_limit,
-            put_admission_wait_timeout,
+            put_admission_policy,
         }
     }
 
@@ -234,10 +410,19 @@ impl ConcurrencyManager {
     #[cfg(test)]
     pub(crate) fn with_put_admission_for_test(enabled: bool, limit: usize, wait_timeout: Duration) -> Self {
         let mut manager = Self::new();
-        manager.put_admission_semaphore = Arc::new(Semaphore::new(if enabled { limit } else { 0 }));
-        manager.put_admission_enabled = enabled;
-        manager.put_admission_limit = limit;
-        manager.put_admission_wait_timeout = wait_timeout;
+        manager.put_admission_policy = PutAdmissionPolicy::strict_for_test(enabled, limit, wait_timeout);
+        manager
+    }
+
+    #[cfg(test)]
+    pub(crate) fn with_large_put_admission_for_test(
+        enabled: bool,
+        limit: usize,
+        min_size_bytes: usize,
+        wait_timeout: Duration,
+    ) -> Self {
+        let mut manager = Self::new();
+        manager.put_admission_policy = PutAdmissionPolicy::large_for_test(enabled, limit, min_size_bytes, wait_timeout);
         manager
     }
 
@@ -326,30 +511,13 @@ impl ConcurrencyManager {
         }
     }
 
-    /// Admit a foreground PutObject request under the experimental fixed-count gate.
+    /// Admit a foreground PutObject request under the configured write gate.
     ///
-    /// The default-off path returns [`PutObjectAdmission::Disabled`] without
-    /// touching the semaphore, preserving legacy behavior. When enabled, the
-    /// permit must be acquired before body ingest and held until the store write
-    /// returns, so saturated foreground writes can fail with `SlowDown` before
-    /// creating visible side effects.
-    pub async fn admit_put_object(&self) -> Result<PutObjectAdmission, tokio::sync::AcquireError> {
-        if !self.put_admission_enabled || self.put_admission_limit == 0 {
-            return Ok(PutObjectAdmission::Disabled);
-        }
-
-        if self.put_admission_wait_timeout.is_zero() {
-            return Ok(match self.put_admission_semaphore.clone().try_acquire_owned() {
-                Ok(permit) => PutObjectAdmission::Admitted(permit),
-                Err(tokio::sync::TryAcquireError::NoPermits) => PutObjectAdmission::Rejected,
-                Err(tokio::sync::TryAcquireError::Closed) => PutObjectAdmission::Rejected,
-            });
-        }
-
-        match tokio::time::timeout(self.put_admission_wait_timeout, self.put_admission_semaphore.clone().acquire_owned()).await {
-            Ok(permit) => Ok(PutObjectAdmission::Admitted(permit?)),
-            Err(_) => Ok(PutObjectAdmission::Rejected),
-        }
+    /// The strict experimental gate applies to every PUT only when explicitly
+    /// enabled. Otherwise the default-on large-object gate protects sustained
+    /// erasure/RPC pressure while keeping small PUTs on the legacy path.
+    pub async fn admit_put_object(&self, size: i64) -> Result<PutObjectAdmission, tokio::sync::AcquireError> {
+        self.put_admission_policy.admit(size).await
     }
 
     // ============================================
@@ -760,35 +928,7 @@ impl ConcurrencyManager {
 
     /// Get a read-only workload admission snapshot for foreground writes.
     pub fn put_object_admission_snapshot(&self) -> WorkloadAdmissionSnapshot {
-        let (active, limit, hard_gate_enabled) = if self.put_admission_enabled && self.put_admission_limit > 0 {
-            (
-                self.put_admission_limit
-                    .saturating_sub(self.put_admission_semaphore.available_permits()),
-                self.put_admission_limit,
-                true,
-            )
-        } else {
-            (PutObjectGuard::concurrent_count(), self.scheduler_config.max_concurrent_reads, false)
-        };
-        let state = if limit == 0 {
-            AdmissionState::Disabled
-        } else if active >= limit {
-            AdmissionState::Saturated
-        } else {
-            AdmissionState::Open
-        };
-
-        let admission =
-            WorkloadAdmissionSnapshot::new(WorkloadClass::ForegroundWrite, state).with_counts(Some(active), None, Some(limit));
-
-        match state {
-            AdmissionState::Disabled => admission.with_reason("foreground write admission disabled"),
-            AdmissionState::Saturated if hard_gate_enabled => {
-                admission.with_reason("foreground write admission permits exhausted")
-            }
-            AdmissionState::Saturated => admission.with_reason("foreground write concurrency reached local pressure limit"),
-            _ => admission,
-        }
+        self.put_admission_policy.snapshot(self.scheduler_config.max_concurrent_reads)
     }
 
     /// Get a read-only workload admission registry snapshot for local storage concurrency.
@@ -862,7 +1002,7 @@ impl Default for ConcurrencyManager {
 mod integration_tests {
     use super::super::io_schedule::{IoLoadLevel, IoPriority};
     use super::super::request_guard::GetObjectGuard;
-    use super::{ConcurrencyManager, PutObjectAdmission};
+    use super::{ConcurrencyManager, PutObjectAdmission, derive_large_put_admission_limit};
     use crate::storage::storage_api::concurrency_consumer::PutObjectGuard;
     use rustfs_concurrency::{AdmissionState, WorkloadAdmissionSnapshotProvider, WorkloadClass};
     use rustfs_io_core::io_profile::{AccessPattern, StorageMedia};
@@ -941,7 +1081,7 @@ mod integration_tests {
     #[serial]
     async fn test_concurrency_manager_workload_admission_snapshot_tracks_put_requests() {
         crate::storage::concurrency::reset_active_put_requests();
-        let manager = ConcurrencyManager::new();
+        let manager = ConcurrencyManager::with_put_admission_for_test(false, 0, Duration::ZERO);
         let initial = manager.put_object_admission_snapshot();
 
         assert_eq!(initial.class, WorkloadClass::ForegroundWrite);
@@ -965,26 +1105,42 @@ mod integration_tests {
         let manager = ConcurrencyManager::with_put_admission_for_test(false, 1, Duration::ZERO);
 
         let admission = manager
-            .admit_put_object()
+            .admit_put_object(1024)
             .await
             .expect("disabled put admission must not close");
 
         assert!(matches!(admission, PutObjectAdmission::Disabled));
-        assert_eq!(manager.put_admission_semaphore.available_permits(), 0);
         assert_eq!(manager.put_object_admission_snapshot().state, AdmissionState::Open);
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn test_concurrency_manager_strict_put_admission_zero_limit_disables_large_gate() {
+        let manager = ConcurrencyManager::with_put_admission_for_test(true, 0, Duration::ZERO);
+
+        let admission = manager
+            .admit_put_object(32 * 1024 * 1024)
+            .await
+            .expect("strict zero-limit put admission must not close");
+
+        assert!(matches!(admission, PutObjectAdmission::Disabled));
+        assert_eq!(manager.put_object_admission_snapshot().state, AdmissionState::Disabled);
+    }
+
     #[tokio::test]
     #[serial]
     async fn test_concurrency_manager_put_admission_rejects_when_limit_full() {
         let manager = ConcurrencyManager::with_put_admission_for_test(true, 1, Duration::ZERO);
 
-        let first = manager.admit_put_object().await.expect("first put admission should acquire");
+        let first = manager
+            .admit_put_object(1024)
+            .await
+            .expect("first put admission should acquire");
         assert!(matches!(first, PutObjectAdmission::Admitted(_)));
         assert_eq!(manager.put_object_admission_snapshot().state, AdmissionState::Saturated);
 
         let second = manager
-            .admit_put_object()
+            .admit_put_object(1024)
             .await
             .expect("full put admission gate should reject, not close");
         assert!(matches!(second, PutObjectAdmission::Rejected));
@@ -995,11 +1151,14 @@ mod integration_tests {
     async fn test_concurrency_manager_put_admission_reuses_released_permit() {
         let manager = ConcurrencyManager::with_put_admission_for_test(true, 1, Duration::ZERO);
 
-        let first = manager.admit_put_object().await.expect("first put admission should acquire");
+        let first = manager
+            .admit_put_object(1024)
+            .await
+            .expect("first put admission should acquire");
         drop(first);
 
         let second = manager
-            .admit_put_object()
+            .admit_put_object(1024)
             .await
             .expect("released put admission permit should be reusable");
         assert!(matches!(second, PutObjectAdmission::Admitted(_)));
@@ -1009,10 +1168,13 @@ mod integration_tests {
     #[serial]
     async fn test_concurrency_manager_put_admission_wait_timeout_rejects() {
         let manager = ConcurrencyManager::with_put_admission_for_test(true, 1, Duration::from_secs(5));
-        let held = manager.admit_put_object().await.expect("first put admission should acquire");
+        let held = manager
+            .admit_put_object(1024)
+            .await
+            .expect("first put admission should acquire");
         let waiter_manager = manager.clone();
 
-        let waiter = tokio::spawn(async move { waiter_manager.admit_put_object().await });
+        let waiter = tokio::spawn(async move { waiter_manager.admit_put_object(1024).await });
         tokio::task::yield_now().await;
         tokio::time::advance(Duration::from_secs(5)).await;
 
@@ -1024,6 +1186,85 @@ mod integration_tests {
         drop(held);
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn test_concurrency_manager_large_put_admission_bypasses_small_puts() {
+        let min_size = rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES;
+        let manager = ConcurrencyManager::with_large_put_admission_for_test(true, 1, min_size, Duration::ZERO);
+
+        let held = manager
+            .admit_put_object(min_size as i64)
+            .await
+            .expect("large put admission should acquire");
+        assert!(matches!(held, PutObjectAdmission::Admitted(_)));
+
+        let small = manager
+            .admit_put_object((min_size - 1) as i64)
+            .await
+            .expect("small put should bypass large admission");
+        assert!(matches!(small, PutObjectAdmission::Disabled));
+
+        let large = manager
+            .admit_put_object(min_size as i64)
+            .await
+            .expect("second large put should reject when the gate is full");
+        assert!(matches!(large, PutObjectAdmission::Rejected));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn test_concurrency_manager_large_put_admission_gates_unknown_size() {
+        let manager = ConcurrencyManager::with_large_put_admission_for_test(true, 1, 32 * 1024 * 1024, Duration::ZERO);
+
+        let held = manager
+            .admit_put_object(-1)
+            .await
+            .expect("unknown-size put admission should acquire");
+        assert!(matches!(held, PutObjectAdmission::Admitted(_)));
+
+        let second = manager
+            .admit_put_object(-1)
+            .await
+            .expect("unknown-size put admission should reject when the gate is full");
+        assert!(matches!(second, PutObjectAdmission::Rejected));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn test_concurrency_manager_large_put_snapshot_tracks_gate() {
+        let manager = ConcurrencyManager::with_large_put_admission_for_test(true, 2, 32 * 1024 * 1024, Duration::ZERO);
+        let first = manager
+            .admit_put_object(32 * 1024 * 1024)
+            .await
+            .expect("first large put admission should acquire");
+        let initial = manager.put_object_admission_snapshot();
+
+        assert_eq!(initial.class, WorkloadClass::ForegroundWrite);
+        assert_eq!(initial.state, AdmissionState::Open);
+        assert_eq!(initial.active, Some(1));
+        assert_eq!(initial.limit, Some(2));
+
+        let second = manager
+            .admit_put_object(32 * 1024 * 1024)
+            .await
+            .expect("second large put admission should acquire");
+        let saturated = manager.put_object_admission_snapshot();
+
+        assert_eq!(saturated.state, AdmissionState::Saturated);
+        assert_eq!(saturated.active, Some(2));
+        assert_eq!(saturated.limit, Some(2));
+        drop((first, second));
+    }
+
+    #[test]
+    fn test_concurrency_manager_derives_large_put_admission_limit_from_scheduler_cap() {
+        assert_eq!(derive_large_put_admission_limit(7, 64), 7);
+        assert_eq!(derive_large_put_admission_limit(0, 64), 32);
+        assert_eq!(derive_large_put_admission_limit(0, 8), 4);
+        assert_eq!(derive_large_put_admission_limit(0, 1), 1);
+        assert_eq!(derive_large_put_admission_limit(0, 0), 32);
+    }
+
     #[tokio::test]
     #[serial]
     async fn test_concurrency_manager_workload_admission_registry_covers_required_classes() {

rustfs/src/storage/sse.rs

@@ -961,9 +961,9 @@ fn sse_kms_key_policy_enforced(principal: Option<&SseKmsPrincipal>) -> bool {
 
 /// Report the configured SSE-KMS authorization mode once, at startup.
 ///
-/// The disabled case warns rather than logs: it is the compatibility default for this
-/// release only, and operators need the lead time to grant the kms actions before the
-/// default flips.
+/// The disabled case warns rather than logs: while enforcement is off, any identity
+/// allowed to write an object can encrypt it under any key, and operators should hear
+/// about that even though disabled is the long-term default.
 pub(crate) fn log_sse_kms_key_policy_mode() {
     if sse_kms_key_policy_enforced(None) {
         tracing::info!(
@@ -984,7 +984,7 @@ pub(crate) fn log_sse_kms_key_policy_mode() {
         "SSE-KMS requests are not authorized against the KMS key they name; any identity allowed to \
          write an object may encrypt it under any key, and any identity allowed to read it may have it \
          decrypted. Grant kms:GenerateDataKey and kms:Decrypt on the keys your workloads use, then set \
-         {ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY}=true. A later release defaults this to enabled."
+         {ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY}=true."
     );
 }
 
@@ -1027,6 +1027,25 @@ async fn authorize_sse_kms_key(
         "Principal is not authorized for the KMS key resolved for this request"
     );
 
+    // One warn per process, not per request: anonymous denials are driven by
+    // unauthenticated traffic, so a per-request warn would let anyone flood the
+    // log. Per-request detail stays on the audit entry and the debug event above.
+    if principal.account.is_empty() {
+        static ANONYMOUS_DENIAL_WARNED: std::sync::Once = std::sync::Once::new();
+        ANONYMOUS_DENIAL_WARNED.call_once(|| {
+            tracing::warn!(
+                component = LOG_COMPONENT_STORAGE,
+                subsystem = LOG_SUBSYSTEM_SSE,
+                event = "sse_kms_anonymous_key_authorization_denied",
+                action = ?action,
+                "Anonymous requests are being denied by SSE-KMS per-key authorization: anonymous \
+                 callers hold no kms grants, so a public bucket serving SSE-KMS objects is \
+                 incompatible with {ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY}=true. Reported once per \
+                 process; per-request denials are on audit entries and at debug level."
+            );
+        });
+    }
+
     Err(ApiError {
         code: S3ErrorCode::AccessDenied,
         message: "Access Denied".to_string(),

rustfs/src/update.rs

@@ -59,8 +59,6 @@ pub struct UpdateCheckResult {
 
 /// Version checker
 pub struct VersionChecker {
-    /// HTTP client
-    client: reqwest::Client,
     /// Version server URL
     version_url: String,
     /// Request timeout
@@ -76,14 +74,7 @@ impl Default for VersionChecker {
 impl VersionChecker {
     /// Create a new version checker
     pub fn new() -> Self {
-        let client = reqwest::Client::builder()
-            .timeout(Duration::from_secs(10))
-            .user_agent(format!("RustFS/{}", get_current_version()))
-            .build()
-            .unwrap_or_else(|_| reqwest::Client::new());
-
         Self {
-            client,
             version_url: "https://version.rustfs.com/latest.json".to_string(),
             timeout: Duration::from_secs(10),
         }
@@ -91,14 +82,7 @@ impl VersionChecker {
 
     /// Create version checker with custom configuration
     pub fn with_config(url: String, timeout: Duration) -> Self {
-        let client = reqwest::Client::builder()
-            .timeout(timeout)
-            .user_agent(format!("RustFS/{}", get_current_version()))
-            .build()
-            .unwrap_or_else(|_| reqwest::Client::new());
-
         Self {
-            client,
             version_url: url,
             timeout,
         }
@@ -108,9 +92,13 @@ impl VersionChecker {
     pub async fn check_for_updates(&self) -> Result<UpdateCheckResult, UpdateCheckError> {
         let current_version = get_current_version();
         debug!("Checking for updates, current version: {}", current_version);
+        let client = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .user_agent(format!("RustFS/{current_version}"))
+            .build()?;
 
         // Send HTTP GET request to get latest version information
-        let response = self.client.get(&self.version_url).timeout(self.timeout).send().await?;
+        let response = client.get(&self.version_url).timeout(self.timeout).send().await?;
 
         if !response.status().is_success() {
             let status = response.status();
@@ -182,6 +170,32 @@ pub async fn check_updates_with_url(url: String) -> Result<UpdateCheckResult, Up
 mod tests {
     use super::*;
 
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn version_checker_construction_does_not_require_system_roots() {
+        #[cfg(target_os = "linux")]
+        {
+            let temp = tempfile::tempdir().expect("temporary certificate directory");
+            let cert_file = temp.path().join("empty.pem");
+            std::fs::write(&cert_file, []).expect("empty certificate file");
+            let cert_file = cert_file.to_string_lossy().into_owned();
+            let result =
+                temp_env::async_with_vars([("SSL_CERT_FILE", Some(cert_file.as_str())), ("SSL_CERT_DIR", Some(""))], async {
+                    let checker = VersionChecker::new();
+                    checker.check_for_updates().await
+                })
+                .await;
+            assert!(matches!(result, Err(UpdateCheckError::HttpError(_))));
+        }
+
+        #[cfg(not(target_os = "linux"))]
+        {
+            let checker = VersionChecker::new();
+            assert_eq!(checker.version_url, "https://version.rustfs.com/latest.json");
+            assert_eq!(checker.timeout, Duration::from_secs(10));
+        }
+    }
+
     #[tokio::test]
     async fn test_get_current_version() {
         let version = get_current_version();

scripts/security/check_preview_release_workflow.sh

@@ -140,6 +140,14 @@ done
 latest_guard="startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')"
 require_job_if "$build_workflow" "update-latest-version" "    if: $latest_guard"
 require_line "$build_workflow" "    needs: [ build-check, publish-release ]" "latest update must follow release publication"
+
+# Preview releases are internal validation artifacts: once the deliverable
+# release is published they are deleted, while their tags stay behind.
+require_job_if "$build_workflow" "cleanup-preview-releases" "    if: $latest_guard"
+require_line "$build_workflow" "            gh release delete \"\$preview_tag\" --yes" "preview release cleanup after publication"
+require_line "$build_workflow" "              | select(.tag_name | startswith(\$tag + \"-preview.\"))" "cleanup must match the target's own preview tags"
+require_line "$build_workflow" "              | select(.tag_name | ltrimstr(\$tag + \"-preview.\") | test(\"^[0-9]+\$\"))" "cleanup must match a numeric preview iteration"
+require_absent "$build_workflow" "--cleanup-tag" "preview tags must survive their release cleanup"
 require_line "$build_workflow" "          TARGET_COMMITISH=\$(git rev-parse --verify \"refs/tags/\${TAG}^{commit}\")" "release target commit resolution"
 require_line "$build_workflow" "          ./scripts/release/create_or_update_release.sh \\" "managed release creation"
 require_absent "$build_workflow" "git tag -l --format='%(contents)'" "annotated tag messages must not become release notes"

scripts/table-catalog/README.md

@@ -163,10 +163,10 @@ python3 scripts/table-catalog/engine_compatibility.py --print-live-evidence-sche
 ```
 
 Use these outputs when updating release notes, PR descriptions, or follow-up
-work items. They are intentionally conservative: only PyIceberg is automated by
-this script today. Spark has a repeatable manual/live harness with pinned
-client package inputs, generated configuration, generated SQL, expected
-results, and a CI opt-in gate. Trino, DuckDB, Databend, and Snowflake now have
+work items. They are intentionally conservative: PyIceberg and DuckDB have
+separate automated smoke entrypoints. Spark has a repeatable manual/live harness
+with pinned client package inputs, generated configuration, generated SQL,
+expected results, and a CI opt-in gate. Trino, Databend, and Snowflake have
 generated manual probe inputs, but they remain opt-in and do not promote write
 or full vendor interoperability claims.
 
@@ -241,6 +241,7 @@ python3 scripts/table-catalog/engine_compatibility.py \
   --table-bucket analytics \
   --print-spark-config
 python3 scripts/table-catalog/engine_compatibility.py --print-spark-sql --cleanup
+python3 scripts/table-catalog/engine_compatibility.py --print-duckdb-rest-sql
 python3 scripts/table-catalog/engine_compatibility.py --print-live-conformance --cleanup
 python3 scripts/table-catalog/engine_compatibility.py --print-operations-guide
 ```
@@ -310,7 +311,7 @@ The smoke test also probes catalog-backed advanced Iceberg surfaces:
 | PyIceberg | Automated smoke target | create namespace, create table, append, reload, scan, metadata-location, refs, views, maintenance, diagnostics, optional catalog-vended table credentials with exact-prefix data-plane scope probe |
 | Spark Iceberg REST catalog | Manual/live harness | pinned Spark and Iceberg package inputs, configuration, SQL, run command, expected row count, and cleanup can be generated for a running RustFS endpoint; CI execution is opt-in |
 | Trino Iceberg REST catalog | Manual/live read probe | generated catalog properties and a read-only SELECT probe for a table created by PyIceberg or Spark; no write compatibility claim yet |
-| DuckDB Iceberg | Manual/live read probe | generated httpfs/iceberg SQL using an operator-supplied current metadata location; read-path only |
+| DuckDB Iceberg | Automated smoke target | metadata-location read plus generic REST Catalog single-table DDL, DML, schema evolution, snapshots, `/iceberg` and `/_iceberg` signing, fail-closed unsupported boundaries, endpoint-disabled non-atomic multi-table mode, concurrent writers, and PyIceberg cross-read |
 | StarRocks Iceberg REST catalog | Documented, not automated | external catalog read-path reference only |
 | Databend | Manual/live S3 stage probe | generated S3 stage read probe for table data files; Iceberg REST catalog integration is not claimed |
 | Snowflake/Open Catalog integrations | Manual reference probe | generated external volume/catalog SQL template; live RustFS interoperability is not claimed |
@@ -503,6 +504,70 @@ RUSTFS_TABLE_CATALOG_CREDENTIAL_TTL_SECONDS=900
 
 The TTL is clamped to the supported short-lived range by the server.
 
+## DuckDB REST Catalog Profile
+
+DuckDB can read an individual Iceberg table with `iceberg_scan` or attach RustFS
+as a generic Iceberg REST Catalog. The metadata-location path remains read-only.
+The attached catalog path is the prerequisite for DuckDB writes.
+
+Generate the canonical RustFS REST Catalog profile:
+
+```bash
+python3 scripts/table-catalog/engine_compatibility.py \
+  --endpoint http://127.0.0.1:9000 \
+  --warehouse rustfs-s3table-smoke \
+  --namespace smoke \
+  --table events \
+  --rest-path /iceberg \
+  --rest-signing-name s3 \
+  --print-duckdb-rest-sql
+```
+
+Generate the compatibility alias profile by changing the last three arguments:
+
+```bash
+python3 scripts/table-catalog/engine_compatibility.py \
+  --rest-path /_iceberg \
+  --rest-signing-name s3tables \
+  --print-duckdb-rest-sql
+```
+
+The generated `ATTACH` disables staged create, post-create metadata updates,
+multi-table commit, client-side file removal, and purge-on-drop. These options
+keep DuckDB within RustFS's claimed single-table REST surface. Do not replace
+the explicit endpoint with DuckDB `ENDPOINT_TYPE S3_TABLES`; that shortcut is
+for AWS S3 Tables endpoint and warehouse shapes.
+
+Run the repeatable DuckDB 1.5.5 smoke against an already running RustFS:
+
+```bash
+python3 scripts/table-catalog/duckdb_smoke.py \
+  --duckdb /path/to/duckdb \
+  --endpoint http://127.0.0.1:9000 \
+  --bucket rustfs-duckdb-smoke \
+  --namespace duckdb_smoke \
+  --table events \
+  --cleanup \
+  --rustfs-build rustfs-v1.0.0-rc.4 \
+  --git-sha "$(git rev-parse HEAD)" \
+  --catalog-backing object \
+  --live-evidence-output /tmp/rustfs-duckdb-live-evidence.json
+```
+
+The script requires the same PyIceberg, PyArrow, and boto3 dependencies as the
+PyIceberg smoke because it verifies both cross-engine directions. It creates an
+isolated namespace, keeps the final verified table at two rows for the shared
+evidence contract, and cleans all smoke tables only when `--cleanup` is set.
+It refuses to remove pre-existing suffixed smoke tables unless `--replace` is
+set explicitly. Cleanup preserves a namespace that existed before the run.
+
+The automated claim is limited to DuckDB 1.5.5, static S3 credentials, and the
+single-table scenarios exercised by this script. It does not claim DuckDB's AWS
+`S3_TABLES` shortcut, staged create, purge-on-drop, format v3, multi-table
+atomicity, or catalog-vended credential integration. The smoke verifies that
+DuckDB can run a two-table transaction with its multi-table commit endpoint
+disabled, but each table remains an independent RustFS commit.
+
 ## Spark Manual/Live Harness
 
 Spark validation should use the same RustFS endpoint and warehouse bucket as the
@@ -591,8 +656,9 @@ engines that are not run by default in RustFS CI:
 - Trino: catalog properties and a read-only `SELECT COUNT(*)` command for a
   table already created by PyIceberg or Spark. Trino write compatibility is not
   claimed.
-- DuckDB: `httpfs` and `iceberg` SQL using an operator-supplied current Iceberg
-  metadata location. DuckDB write and commit compatibility are not claimed.
+- DuckDB: a legacy `httpfs` and `iceberg` read probe using an operator-supplied
+  current Iceberg metadata location. The separate `duckdb_smoke.py` entrypoint
+  owns the automated generic REST Catalog single-table read/write claim.
 - Databend: an S3 stage read probe for Parquet data files under the table
   warehouse. Databend Iceberg REST Catalog integration is not claimed.
 - Snowflake: an operator-adapted external volume/catalog integration SQL

scripts/table-catalog/duckdb_smoke.py

@@ -0,0 +1,653 @@
+#!/usr/bin/env python3
+"""DuckDB Iceberg REST Catalog smoke test for RustFS S3 Tables."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import time
+import urllib.parse
+from concurrent.futures import ThreadPoolExecutor
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+import engine_compatibility
+import pyiceberg_smoke
+
+
+DEFAULT_DUCKDB_VERSION = engine_compatibility.DEFAULT_DUCKDB_VERSION
+IDENTIFIER_RE = re.compile(r"^[A-Za-z][A-Za-z0-9_]{0,47}$")
+
+
+@dataclass(frozen=True)
+class DuckDBExecution:
+    returncode: int
+    batches: list[list[dict[str, Any]]]
+    stdout: str
+    stderr: str
+
+
+@dataclass(frozen=True)
+class DuckDBSmokeResult:
+    client_version: str
+    metadata_location: str
+    row_count: int
+    cleanup_result: str
+    checks: dict[str, str]
+
+
+def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
+    run_id = str(int(time.time()))
+    parser = argparse.ArgumentParser(description="Run DuckDB Iceberg REST Catalog conformance against RustFS.")
+    parser.add_argument("--endpoint", default=os.getenv("RUSTFS_ENDPOINT", "http://127.0.0.1:9000"))
+    parser.add_argument("--access-key", default=os.getenv("RUSTFS_ACCESS_KEY", "rustfsadmin"))
+    parser.add_argument("--secret-key", default=os.getenv("RUSTFS_SECRET_KEY", "rustfsadmin"))
+    parser.add_argument("--region", default=os.getenv("RUSTFS_REGION", "us-east-1"))
+    parser.add_argument("--bucket", default=os.getenv("RUSTFS_TABLE_BUCKET", "rustfs-duckdb-smoke"))
+    parser.add_argument("--namespace", default=os.getenv("RUSTFS_TABLE_NAMESPACE", f"duckdb_smoke_{run_id}"))
+    parser.add_argument("--table", default=os.getenv("RUSTFS_TABLE_NAME", "events"))
+    parser.add_argument("--duckdb", default=os.getenv("DUCKDB_BIN", "duckdb"))
+    parser.add_argument("--duckdb-version", default=DEFAULT_DUCKDB_VERSION)
+    parser.add_argument("--timeout", type=float, default=float(os.getenv("RUSTFS_TABLE_SMOKE_TIMEOUT", "60")))
+    parser.add_argument("--cleanup", action="store_true")
+    parser.add_argument("--replace", action="store_true", help="Drop existing smoke tables with matching identifiers first.")
+    parser.add_argument("--insecure", action="store_true")
+    parser.add_argument("--live-evidence-output")
+    parser.add_argument("--rustfs-build", default=os.getenv("RUSTFS_BUILD", "operator-recorded"))
+    parser.add_argument("--git-sha", default=os.getenv("RUSTFS_GIT_SHA", "operator-recorded"))
+    parser.add_argument("--catalog-backing", default=os.getenv("RUSTFS_TABLE_CATALOG_BACKING", "operator-recorded"))
+    parser.add_argument("--operator", default=os.getenv("USER", "operator-recorded"))
+    parser.add_argument("--run-timestamp-utc")
+    args = parser.parse_args(argv)
+    for label, value in [("namespace", args.namespace), ("table", args.table)]:
+        if not IDENTIFIER_RE.fullmatch(value):
+            parser.error(f"{label} must start with a letter and contain at most 48 ASCII letters, digits, or underscores")
+    return args
+
+
+def duckdb_path(value: str) -> str:
+    resolved = shutil.which(value)
+    if resolved is None:
+        raise RuntimeError(f"DuckDB executable was not found: {value}")
+    return resolved
+
+
+def duckdb_client_version(executable: str, timeout: float) -> str:
+    process = subprocess.run(
+        [executable, "-csv", "-noheader", "-c", "SELECT version();"],
+        capture_output=True,
+        text=True,
+        timeout=timeout,
+        check=False,
+    )
+    if process.returncode != 0:
+        raise RuntimeError(f"DuckDB version probe failed: {process.stderr.strip()}")
+    version = process.stdout.strip().removeprefix("v")
+    if not version:
+        raise RuntimeError("DuckDB version probe returned an empty version")
+    return version
+
+
+def parse_duckdb_json(stdout: str) -> list[list[dict[str, Any]]]:
+    batches: list[list[dict[str, Any]]] = []
+    decoder = json.JSONDecoder()
+    offset = 0
+    while offset < len(stdout):
+        while offset < len(stdout) and stdout[offset].isspace():
+            offset += 1
+        if offset == len(stdout):
+            break
+        value, offset = decoder.raw_decode(stdout, offset)
+        if not isinstance(value, list) or any(not isinstance(row, dict) for row in value):
+            raise RuntimeError("DuckDB JSON output did not contain row objects")
+        batches.append(value)
+    return batches
+
+
+def run_duckdb(executable: str, sql: str, timeout: float) -> DuckDBExecution:
+    process = subprocess.run(
+        [executable, "-json", "-c", sql],
+        capture_output=True,
+        text=True,
+        timeout=timeout,
+        check=False,
+    )
+    batches = parse_duckdb_json(process.stdout) if process.returncode == 0 else []
+    return DuckDBExecution(process.returncode, batches, process.stdout, process.stderr)
+
+
+def require_duckdb_success(execution: DuckDBExecution, label: str) -> None:
+    if execution.returncode != 0:
+        message = execution.stderr.strip() or execution.stdout.strip()
+        raise RuntimeError(f"DuckDB {label} failed: {message}")
+
+
+def require_duckdb_error(execution: DuckDBExecution, label: str, expected: str) -> None:
+    if execution.returncode == 0:
+        raise RuntimeError(f"DuckDB {label} unexpectedly succeeded")
+    message = f"{execution.stdout}\n{execution.stderr}"
+    if expected not in message:
+        raise RuntimeError(f"DuckDB {label} failed without expected error {expected!r}: {message.strip()}")
+
+
+def batches_with_column(execution: DuckDBExecution, column: str) -> list[list[dict[str, Any]]]:
+    return [batch for batch in execution.batches if batch and column in batch[0]]
+
+
+def table_name(base: str, suffix: str) -> str:
+    return f"{base}_{suffix}"
+
+
+def table_identifier(catalog: str, namespace: str, table: str) -> str:
+    return ".".join(
+        [
+            engine_compatibility.quote_double_identifier(catalog),
+            engine_compatibility.quote_double_identifier(namespace),
+            engine_compatibility.quote_double_identifier(table),
+        ]
+    )
+
+
+def profile_sql(
+    args: argparse.Namespace,
+    *,
+    catalog: str,
+    table: str,
+    rest_path: str = "/iceberg",
+    signing_name: str = "s3",
+) -> str:
+    return engine_compatibility.duckdb_rest_catalog_sql(
+        endpoint=args.endpoint,
+        warehouse=args.bucket,
+        access_key=args.access_key,
+        secret_key=args.secret_key,
+        region=args.region,
+        catalog_name=catalog,
+        namespace=args.namespace,
+        table=table,
+        rest_path=rest_path,
+        rest_signing_name=signing_name,
+    )
+
+
+def attach_sql(
+    args: argparse.Namespace,
+    *,
+    catalog: str,
+    rest_path: str,
+    signing_name: str,
+    compatibility_options: bool,
+    purge_requested: bool = False,
+) -> str:
+    options = [
+        "  TYPE iceberg",
+        f"  ENDPOINT {engine_compatibility.sql_string(f'{args.endpoint.rstrip('/')}{rest_path}')}",
+        "  AUTHORIZATION_TYPE 'sigv4'",
+        "  SECRET 'rustfs_s3'",
+        f"  SIGV4_REGION {engine_compatibility.sql_string(args.region)}",
+        f"  SIGV4_SERVICE {engine_compatibility.sql_string(signing_name)}",
+        "  ACCESS_DELEGATION_MODE 'none'",
+    ]
+    if compatibility_options:
+        options.extend(
+            [
+                "  STAGE_CREATE_TABLES false",
+                "  SKIP_CREATE_TABLE_METADATA_UPDATES true",
+                "  DISABLE_MULTI_TABLE_COMMIT true",
+                "  REMOVE_FILES_ON_DELETE false",
+                f"  PURGE_REQUESTED {'true' if purge_requested else 'false'}",
+                "  SUPPORT_NESTED_NAMESPACES false",
+            ]
+        )
+    rendered_options = ",\n".join(options)
+    return (
+        f"ATTACH {engine_compatibility.sql_string(args.bucket)} "
+        f"AS {engine_compatibility.quote_double_identifier(catalog)} (\n{rendered_options}\n);\n"
+    )
+
+
+def canonical_positive_sql(args: argparse.Namespace, seed_table: str, write_table: str, purge_table: str, drop_table: str) -> str:
+    catalog = "rustfs_duckdb"
+    namespace = ".".join(
+        [
+            engine_compatibility.quote_double_identifier(catalog),
+            engine_compatibility.quote_double_identifier(args.namespace),
+        ]
+    )
+    write_identifier = table_identifier(catalog, args.namespace, write_table)
+    purge_identifier = table_identifier(catalog, args.namespace, purge_table)
+    drop_identifier = table_identifier(catalog, args.namespace, drop_table)
+    return profile_sql(args, catalog=catalog, table=seed_table) + "\n".join(
+        [
+            f"CREATE SCHEMA IF NOT EXISTS {namespace};",
+            f"CREATE TABLE {write_identifier} (id BIGINT, payload VARCHAR);",
+            f"INSERT INTO {write_identifier} VALUES (10, 'ten'), (20, 'twenty');",
+            f"UPDATE {write_identifier} SET payload = 'TWENTY' WHERE id = 20;",
+            f"DELETE FROM {write_identifier} WHERE id = 10;",
+            f"ALTER TABLE {write_identifier} ADD COLUMN category VARCHAR;",
+            f"INSERT INTO {write_identifier} VALUES (30, 'thirty', 'new');",
+            f"MERGE INTO {write_identifier} AS target",
+            "USING (VALUES (20, 'twenty-merged', 'merged'), (40, 'forty', 'inserted')) AS source(id, payload, category)",
+            "ON target.id = source.id",
+            "WHEN MATCHED THEN UPDATE SET payload = source.payload, category = source.category",
+            "WHEN NOT MATCHED THEN INSERT (id, payload, category) VALUES (source.id, source.payload, source.category);",
+            f"DELETE FROM {write_identifier} WHERE id = 30;",
+            f"SELECT id, payload, category FROM {write_identifier} ORDER BY id;",
+            f"SELECT count(*) AS snapshot_count FROM iceberg_snapshots({write_identifier});",
+            f"CREATE TABLE {drop_identifier} (id BIGINT);",
+            f"INSERT INTO {drop_identifier} VALUES (1);",
+            f"DROP TABLE {drop_identifier};",
+            f"CREATE TABLE {purge_identifier} (id BIGINT);",
+            f"INSERT INTO {purge_identifier} VALUES (1);",
+            f"SELECT count(*) AS row_count FROM {write_identifier};",
+        ]
+    ) + "\n"
+
+
+def alias_sql(args: argparse.Namespace, write_table: str) -> str:
+    catalog = "rustfs_compat"
+    identifier = table_identifier(catalog, args.namespace, write_table)
+    return profile_sql(args, catalog=catalog, table=write_table, rest_path="/_iceberg", signing_name="s3tables") + "\n".join(
+        [
+            f"INSERT INTO {identifier} VALUES (50, 'fifty', 'compat');",
+            f"SELECT count(*) AS alias_row_count FROM {identifier};",
+            f"DELETE FROM {identifier} WHERE id = 50;",
+            f"SELECT count(*) AS alias_final_row_count FROM {identifier};",
+        ]
+    ) + "\n"
+
+
+def concurrent_insert_sql(args: argparse.Namespace, catalog: str, write_table: str, row_id: int) -> str:
+    identifier = table_identifier(catalog, args.namespace, write_table)
+    return profile_sql(args, catalog=catalog, table=write_table) + f"INSERT INTO {identifier} VALUES ({row_id}, 'writer-{row_id}', 'concurrent');\n"
+
+
+def multi_table_sql(args: argparse.Namespace, seed_table: str, write_table: str, purge_table: str) -> str:
+    catalog = "multi_table"
+    first = table_identifier(catalog, args.namespace, write_table)
+    second = table_identifier(catalog, args.namespace, purge_table)
+    return profile_sql(args, catalog=catalog, table=seed_table) + "\n".join(
+        [
+            "BEGIN TRANSACTION;",
+            f"INSERT INTO {first} VALUES (999, 'multi-a', 'non-atomic');",
+            f"INSERT INTO {second} VALUES (999);",
+            "COMMIT;",
+        ]
+    ) + "\n"
+
+
+def negative_sql(args: argparse.Namespace, *, kind: str, seed_table: str, write_table: str, purge_table: str) -> str:
+    bootstrap = f"bootstrap_{kind}"
+    sql = profile_sql(args, catalog=bootstrap, table=seed_table)
+    sql += f"DETACH {engine_compatibility.quote_double_identifier(bootstrap)};\n"
+    if kind == "stage-create":
+        catalog = "stage_default"
+        sql += attach_sql(
+            args,
+            catalog=catalog,
+            rest_path="/iceberg",
+            signing_name="s3",
+            compatibility_options=False,
+        )
+        sql += f"CREATE TABLE {table_identifier(catalog, args.namespace, table_name(args.table, 'stage'))} (id BIGINT);\n"
+        return sql
+    if kind == "purge":
+        catalog = "purge_requested"
+        sql += attach_sql(
+            args,
+            catalog=catalog,
+            rest_path="/iceberg",
+            signing_name="s3",
+            compatibility_options=True,
+            purge_requested=True,
+        )
+        sql += f"DROP TABLE {table_identifier(catalog, args.namespace, purge_table)};\n"
+        return sql
+    if kind == "format-v3":
+        catalog = "format_v3"
+        sql += attach_sql(
+            args,
+            catalog=catalog,
+            rest_path="/iceberg",
+            signing_name="s3",
+            compatibility_options=True,
+        )
+        identifier = table_identifier(catalog, args.namespace, table_name(args.table, "v3"))
+        sql += f"CREATE TABLE {identifier} (id BIGINT) WITH ('format-version' = '3');\n"
+        return sql
+    raise ValueError(f"unknown negative DuckDB smoke kind: {kind}")
+
+
+def pyiceberg_args(args: argparse.Namespace) -> argparse.Namespace:
+    return argparse.Namespace(
+        profile="rustfs",
+        endpoint=args.endpoint,
+        access_key=args.access_key,
+        secret_key=args.secret_key,
+        region=args.region,
+        bucket=args.bucket,
+        warehouse=None,
+        table_bucket=None,
+        account_id="000000000000",
+        warehouse_name=None,
+        catalog_uri=None,
+        namespace=args.namespace,
+        table=args.table,
+        catalog_name="rustfs_duckdb_pyiceberg",
+        rest_path="/iceberg",
+        rest_signing_name="s3",
+        require_vended_credentials=False,
+        timeout=args.timeout,
+        insecure=args.insecure,
+    )
+
+
+def prepare_smoke_tables(catalog: Any, namespace: str, tables: list[str], replace: bool) -> None:
+    existing = [table for table in tables if pyiceberg_smoke.table_exists(catalog, (namespace, table))]
+    if existing and not replace:
+        identifiers = ", ".join(f"{namespace}.{table}" for table in existing)
+        raise RuntimeError(f"DuckDB smoke tables already exist: {identifiers}; rerun with --replace to remove them")
+    for table in existing:
+        catalog.drop_table((namespace, table))
+
+
+def seed_pyiceberg_table(catalog: Any, args: argparse.Namespace, deps: pyiceberg_smoke.RuntimeDeps, table: str) -> None:
+    identifier = (args.namespace, table)
+    schema = deps.pyarrow.schema(
+        [
+            deps.pyarrow.field("id", deps.pyarrow.int64(), nullable=False),
+            deps.pyarrow.field("payload", deps.pyarrow.string(), nullable=False),
+        ]
+    )
+    created = catalog.create_table(identifier, schema=schema)
+    created.append(
+        deps.pyarrow.Table.from_pylist(
+            [{"id": 1, "payload": "alpha"}, {"id": 2, "payload": "beta"}],
+            schema=schema,
+        )
+    )
+
+
+def pyiceberg_rows(catalog: Any, namespace: str, table: str) -> list[dict[str, Any]]:
+    rows = catalog.load_table((namespace, table)).scan().to_arrow().to_pylist()
+    return sorted(rows, key=lambda row: row["id"])
+
+
+def run_concurrent_inserts(executable: str, args: argparse.Namespace, write_table: str) -> str:
+    probes = [("writer_a", 60), ("writer_b", 70)]
+    with ThreadPoolExecutor(max_workers=2) as executor:
+        futures = [
+            executor.submit(run_duckdb, executable, concurrent_insert_sql(args, catalog, write_table, row_id), args.timeout)
+            for catalog, row_id in probes
+        ]
+    executions = [future.result() for future in futures]
+    retried = False
+    for (catalog, row_id), execution in zip(probes, executions, strict=True):
+        if execution.returncode == 0:
+            continue
+        error_text = f"{execution.stdout}\n{execution.stderr}".lower()
+        if not any(marker in error_text for marker in ["409", "conflict", "version token"]):
+            require_duckdb_success(execution, f"concurrent writer {row_id}")
+        retried = True
+        retry = run_duckdb(executable, concurrent_insert_sql(args, f"{catalog}_retry", write_table, row_id), args.timeout)
+        require_duckdb_success(retry, f"concurrent writer retry {row_id}")
+    return "passed-with-serial-retry" if retried else "passed-concurrently"
+
+
+def cleanup_tables(catalog: Any, namespace: str, tables: list[str], *, drop_namespace: bool) -> str:
+    cleanup_errors: list[str] = []
+    for table in tables:
+        try:
+            pyiceberg_smoke.drop_table_if_present(catalog, (namespace, table))
+        except Exception as error:
+            cleanup_errors.append(f"{table}: {error}")
+    if drop_namespace:
+        try:
+            catalog.drop_namespace(namespace)
+        except Exception as error:
+            cleanup_errors.append(f"namespace: {error}")
+    if cleanup_errors:
+        raise RuntimeError("DuckDB smoke cleanup failed: " + "; ".join(cleanup_errors))
+    return "dropped-tables-and-namespace" if drop_namespace else "dropped-tables-preserved-existing-namespace"
+
+
+def run_smoke(args: argparse.Namespace, deps: pyiceberg_smoke.RuntimeDeps) -> DuckDBSmokeResult:
+    executable = duckdb_path(args.duckdb)
+    client_version = duckdb_client_version(executable, args.timeout)
+    if client_version != args.duckdb_version:
+        raise RuntimeError(f"expected DuckDB {args.duckdb_version}, found {client_version}")
+
+    endpoint = pyiceberg_smoke.normalized_endpoint(args.endpoint)
+    pyiceberg_smoke.ensure_local_proxy_bypass(endpoint)
+    pyiceberg_smoke.ensure_aws_env(args.access_key, args.secret_key, args.region)
+    iceberg_args = pyiceberg_args(args)
+    pyiceberg_smoke.ensure_bucket(iceberg_args, deps)
+    pyiceberg_smoke.enable_table_bucket(iceberg_args, deps)
+
+    seed_table = table_name(args.table, "seed")
+    write_table = table_name(args.table, "write")
+    purge_table = table_name(args.table, "purge")
+    drop_table = table_name(args.table, "drop")
+    stage_table = table_name(args.table, "stage")
+    v3_table = table_name(args.table, "v3")
+    smoke_tables = [seed_table, write_table, purge_table, drop_table, stage_table, v3_table]
+    catalog = deps.load_catalog(iceberg_args.catalog_name, **pyiceberg_smoke.catalog_properties(iceberg_args))
+    pyiceberg_smoke.install_rustfs_rest_sigv4_adapter(catalog, iceberg_args, deps)
+    namespace_preexisting = bool(catalog.namespace_exists(args.namespace))
+    prepare_smoke_tables(catalog, args.namespace, smoke_tables, args.replace)
+    pyiceberg_smoke.ensure_namespace(catalog, args.namespace)
+    seed_pyiceberg_table(catalog, args, deps, seed_table)
+
+    checks: dict[str, str] = {}
+    cleanup_result = "not-requested"
+    metadata_location = "operator-recorded"
+    try:
+        positive = run_duckdb(
+            executable,
+            canonical_positive_sql(args, seed_table, write_table, purge_table, drop_table),
+            args.timeout,
+        )
+        require_duckdb_success(positive, "canonical REST catalog lifecycle")
+        row_count_batches = batches_with_column(positive, "row_count")
+        if len(row_count_batches) < 2 or row_count_batches[0][0]["row_count"] != 2 or row_count_batches[-1][0]["row_count"] != 2:
+            raise RuntimeError("DuckDB canonical REST catalog row counts did not remain at 2")
+        result_batches = batches_with_column(positive, "id")
+        expected_rows = [
+            {"id": 20, "payload": "twenty-merged", "category": "merged"},
+            {"id": 40, "payload": "forty", "category": "inserted"},
+        ]
+        if not result_batches or result_batches[-1] != expected_rows:
+            raise RuntimeError(f"DuckDB canonical DML returned unexpected rows: {result_batches[-1] if result_batches else []}")
+        snapshot_batches = batches_with_column(positive, "snapshot_count")
+        if not snapshot_batches or snapshot_batches[-1][0]["snapshot_count"] < 1:
+            raise RuntimeError("DuckDB snapshot metadata probe returned no snapshots")
+        if catalog.table_exists((args.namespace, drop_table)):
+            raise RuntimeError("DuckDB DROP TABLE did not remove the catalog entry")
+        checks["canonical_rest_catalog"] = "pass"
+        checks["single_table_ddl_dml"] = "pass"
+        checks["schema_evolution"] = "pass"
+        checks["snapshot_metadata"] = "pass"
+
+        if pyiceberg_rows(catalog, args.namespace, write_table) != expected_rows:
+            raise RuntimeError("PyIceberg did not observe DuckDB-created table rows")
+        checks["pyiceberg_cross_read"] = "pass"
+
+        alias = run_duckdb(executable, alias_sql(args, write_table), args.timeout)
+        require_duckdb_success(alias, "s3tables compatibility alias")
+        alias_counts = batches_with_column(alias, "alias_row_count")
+        alias_final_counts = batches_with_column(alias, "alias_final_row_count")
+        if not alias_counts or alias_counts[-1][0]["alias_row_count"] != 3:
+            raise RuntimeError("DuckDB compatibility alias insert did not produce row_count=3")
+        if not alias_final_counts or alias_final_counts[-1][0]["alias_final_row_count"] != 2:
+            raise RuntimeError("DuckDB compatibility alias cleanup did not restore row_count=2")
+        checks["s3tables_alias"] = "pass"
+
+        negatives = [
+            ("stage-create", "stage-create is not supported"),
+            ("purge", "purgeRequested=true is not supported"),
+            ("format-v3", "unsupported Iceberg table format-version: 3"),
+        ]
+        for kind, expected_error in negatives:
+            execution = run_duckdb(
+                executable,
+                negative_sql(
+                    args,
+                    kind=kind,
+                    seed_table=seed_table,
+                    write_table=write_table,
+                    purge_table=purge_table,
+                ),
+                args.timeout,
+            )
+            require_duckdb_error(execution, kind, expected_error)
+            checks[kind] = "failed-closed"
+        if catalog.table_exists((args.namespace, stage_table)) or catalog.table_exists((args.namespace, v3_table)):
+            raise RuntimeError("a failed DuckDB create probe left a catalog table behind")
+        if not catalog.table_exists((args.namespace, purge_table)):
+            raise RuntimeError("purgeRequested=true removed a table despite the expected failure")
+
+        multi_table = run_duckdb(
+            executable,
+            multi_table_sql(args, seed_table, write_table, purge_table),
+            args.timeout,
+        )
+        require_duckdb_success(multi_table, "multi-table endpoint-disabled mode")
+        if not any(row["id"] == 999 for row in pyiceberg_rows(catalog, args.namespace, write_table)):
+            raise RuntimeError("DuckDB multi-table endpoint-disabled mode did not commit the first table")
+        if not any(row["id"] == 999 for row in pyiceberg_rows(catalog, args.namespace, purge_table)):
+            raise RuntimeError("DuckDB multi-table endpoint-disabled mode did not commit the second table")
+        multi_cleanup_catalog = "cleanup_multi_table"
+        multi_cleanup = profile_sql(args, catalog=multi_cleanup_catalog, table=seed_table) + "\n".join(
+            [
+                f"DELETE FROM {table_identifier(multi_cleanup_catalog, args.namespace, write_table)} WHERE id = 999;",
+                f"DELETE FROM {table_identifier(multi_cleanup_catalog, args.namespace, purge_table)} WHERE id = 999;",
+            ]
+        )
+        require_duckdb_success(
+            run_duckdb(executable, multi_cleanup, args.timeout),
+            "multi-table endpoint-disabled cleanup",
+        )
+        checks["multi_table_endpoint_disabled"] = "pass-single-table-atomicity-only"
+
+        checks["concurrent_writers"] = run_concurrent_inserts(executable, args, write_table)
+        concurrent_rows = pyiceberg_rows(catalog, args.namespace, write_table)
+        if [row["id"] for row in concurrent_rows] != [20, 40, 60, 70]:
+            raise RuntimeError(f"concurrent DuckDB writers produced unexpected rows: {concurrent_rows}")
+        cleanup_catalog = "cleanup_concurrency"
+        cleanup_identifier = table_identifier(cleanup_catalog, args.namespace, write_table)
+        cleanup_sql = profile_sql(args, catalog=cleanup_catalog, table=write_table) + "\n".join(
+            [
+                f"DELETE FROM {cleanup_identifier} WHERE id IN (60, 70);",
+                f"SELECT count(*) AS final_row_count FROM {cleanup_identifier};",
+            ]
+        )
+        cleanup_execution = run_duckdb(executable, cleanup_sql, args.timeout)
+        require_duckdb_success(cleanup_execution, "concurrency cleanup")
+        final_batches = batches_with_column(cleanup_execution, "final_row_count")
+        if not final_batches or final_batches[-1][0]["final_row_count"] != 2:
+            raise RuntimeError("DuckDB concurrency cleanup did not restore row_count=2")
+
+        final_table = catalog.load_table((args.namespace, write_table))
+        final_rows = sorted(final_table.scan().to_arrow().to_pylist(), key=lambda row: row["id"])
+        if final_rows != expected_rows:
+            raise RuntimeError(f"final PyIceberg cross-read returned unexpected rows: {final_rows}")
+        metadata_location = pyiceberg_smoke.table_metadata_location(final_table) or "operator-recorded"
+        if metadata_location == "operator-recorded":
+            response = pyiceberg_smoke.signed_rest_request(
+                argparse.Namespace(**{**vars(iceberg_args), "table": write_table}),
+                deps,
+                "GET",
+                f"/iceberg/v1/{urllib.parse.quote(args.bucket, safe='')}/namespaces/"
+                f"{urllib.parse.quote(args.namespace, safe='')}/tables/{urllib.parse.quote(write_table, safe='')}",
+            )
+            metadata_location = response.get("metadata-location", "operator-recorded")
+        if metadata_location == "operator-recorded":
+            raise RuntimeError("DuckDB smoke could not resolve the final metadata location")
+        metadata_scan = run_duckdb(
+            executable,
+            engine_compatibility.duckdb_sql_probe(
+                endpoint=args.endpoint,
+                access_key=args.access_key,
+                secret_key=args.secret_key,
+                region=args.region,
+                metadata_location=metadata_location,
+            ),
+            args.timeout,
+        )
+        require_duckdb_success(metadata_scan, "metadata-location scan")
+        metadata_counts = batches_with_column(metadata_scan, "row_count")
+        if not metadata_counts or metadata_counts[-1][0]["row_count"] != 2:
+            raise RuntimeError("DuckDB metadata-location scan did not return row_count=2")
+        checks["metadata_location_scan"] = "pass"
+    finally:
+        if args.cleanup:
+            cleanup_result = cleanup_tables(
+                catalog,
+                args.namespace,
+                smoke_tables,
+                drop_namespace=not namespace_preexisting,
+            )
+
+    return DuckDBSmokeResult(client_version, metadata_location, 2, cleanup_result, checks)
+
+
+def current_utc_timestamp() -> str:
+    return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
+
+
+def write_live_evidence(args: argparse.Namespace, result: DuckDBSmokeResult) -> None:
+    if not args.live_evidence_output:
+        return
+    command = pyiceberg_smoke.redacted_command(sys.argv)
+    record = engine_compatibility.live_conformance_evidence_record(
+        client_name="DuckDB Iceberg",
+        client_version=result.client_version,
+        scenario="rest-catalog-single-table-read-write-cross-engine-negative-boundaries",
+        rustfs_build=args.rustfs_build,
+        git_sha=args.git_sha,
+        catalog_backing=args.catalog_backing,
+        endpoint=args.endpoint,
+        warehouse=args.bucket,
+        rest_path="/iceberg",
+        namespace=args.namespace,
+        table=table_name(args.table, "write"),
+        metadata_location=result.metadata_location,
+        run_timestamp_utc=args.run_timestamp_utc or current_utc_timestamp(),
+        operator=args.operator,
+        expected_status="pass",
+        observed_status="pass",
+        row_count=result.row_count,
+        cleanup_result=result.cleanup_result,
+        claim="automated-rest-catalog-smoke",
+        command=command,
+    )
+    document = {
+        "live_conformance_evidence": record,
+        "checks": result.checks,
+        "validation": engine_compatibility.validate_live_conformance_evidence(record),
+    }
+    Path(args.live_evidence_output).write_text(json.dumps(document, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+
+
+def main() -> int:
+    args = parse_args()
+    try:
+        deps = pyiceberg_smoke.load_runtime_deps()
+        result = run_smoke(args, deps)
+        write_live_evidence(args, result)
+        print(json.dumps({"status": "pass", "row_count": result.row_count, "checks": result.checks}, sort_keys=True))
+        return 0
+    except Exception as error:
+        print(f"ERROR: {error}", file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/table-catalog/engine_compatibility.py

@@ -17,7 +17,7 @@ DEFAULT_SPARK_VERSION = "3.5.4"
 DEFAULT_ICEBERG_VERSION = "1.7.1"
 DEFAULT_SCALA_VERSION = "2.12"
 DEFAULT_TRINO_VERSION = "477"
-DEFAULT_DUCKDB_VERSION = "1.3.2"
+DEFAULT_DUCKDB_VERSION = "1.5.5"
 DEFAULT_SNOWFLAKE_CLIENT_VERSION = "operator-recorded"
 DEFAULT_DATABEND_VERSION = "operator-recorded"
 DEFAULT_TRINO_SERVER = "http://127.0.0.1:8080"
@@ -53,7 +53,7 @@ LIVE_EVIDENCE_ALLOWED_CLAIMS = OrderedDict(
         ("PyIceberg", ["automated-smoke"]),
         ("Spark Iceberg REST catalog", ["manual-live-verified"]),
         ("Trino Iceberg REST catalog", ["manual-live-read-verified"]),
-        ("DuckDB Iceberg", ["manual-live-read-verified"]),
+        ("DuckDB Iceberg", ["manual-live-read-verified", "automated-rest-catalog-smoke"]),
         ("Databend", ["manual-live-s3-stage-verified"]),
         ("Snowflake Open Catalog / Iceberg integrations", ["reference-only"]),
     ]
@@ -155,12 +155,15 @@ def engine_compatibility_matrix() -> list[dict[str, Any]]:
         },
         {
             "client": "DuckDB Iceberg",
-            "status": "manual-live-read-probe",
-            "entrypoint": "scripts/table-catalog/engine_compatibility.py --print-live-conformance",
+            "status": "automated-smoke",
+            "entrypoint": "scripts/table-catalog/duckdb_smoke.py",
             "scenarios": [
-                scenario("metadata-read", "manual-live-probe", "read a supplied Iceberg metadata location through DuckDB iceberg_scan"),
-                scenario("read-table", "manual-live-probe", "read-path verification only"),
-                scenario("write-table", "not-claimed", "DuckDB write/commit compatibility is not claimed"),
+                scenario("metadata-read", "automated", "read the final metadata location through DuckDB iceberg_scan"),
+                scenario("catalog-attach", "automated", "attach `/iceberg` with s3 signing and `/_iceberg` with s3tables signing"),
+                scenario("read-table", "automated", "read a PyIceberg-created table through the attached catalog"),
+                scenario("write-table", "automated", "exercise single-table DDL, DML, schema evolution, snapshots, and PyIceberg cross-read"),
+                scenario("unsupported-boundaries", "automated", "verify staged create, purge, and format v3 fail closed"),
+                scenario("multi-table-mode", "automated", "verify DuckDB can avoid the multi-table commit endpoint without claiming cross-table atomicity"),
             ],
         },
         {
@@ -489,8 +492,66 @@ def duckdb_sql_probe(
     return "\n".join(statements) + "\n"
 
 
-def duckdb_command() -> str:
-    return shell_join(["duckdb", "-c", ".read /tmp/rustfs-s3tables-duckdb-read.sql"])
+def duckdb_rest_catalog_sql(
+    *,
+    endpoint: str,
+    warehouse: str,
+    access_key: str,
+    secret_key: str,
+    region: str,
+    catalog_name: str,
+    namespace: str,
+    table: str,
+    rest_path: str,
+    rest_signing_name: str,
+) -> str:
+    parsed = re.match(r"^(https?)://(.+)$", normalized_endpoint(endpoint))
+    if not parsed:
+        raise ValueError("DuckDB REST catalog endpoint must include http:// or https://")
+    scheme, endpoint_without_scheme = parsed.groups()
+    rest_path = normalized_rest_path(rest_path)
+    catalog_identifier = quote_double_identifier(catalog_name)
+    table_identifier = ".".join(
+        [catalog_identifier, quote_double_identifier(namespace), quote_double_identifier(table)]
+    )
+    statements = [
+        "INSTALL httpfs;",
+        "LOAD httpfs;",
+        "INSTALL iceberg;",
+        "LOAD iceberg;",
+        "CREATE OR REPLACE SECRET rustfs_s3 (",
+        "  TYPE s3,",
+        "  PROVIDER config,",
+        f"  KEY_ID {sql_string(access_key)},",
+        f"  SECRET {sql_string(secret_key)},",
+        f"  REGION {sql_string(region)},",
+        f"  ENDPOINT {sql_string(endpoint_without_scheme)},",
+        "  URL_STYLE 'path',",
+        f"  USE_SSL {'true' if scheme == 'https' else 'false'},",
+        f"  SCOPE {sql_string(f's3://{warehouse}')}",
+        ");",
+        f"ATTACH {sql_string(warehouse)} AS {catalog_identifier} (",
+        "  TYPE iceberg,",
+        f"  ENDPOINT {sql_string(f'{normalized_endpoint(endpoint)}{rest_path}')},",
+        "  AUTHORIZATION_TYPE 'sigv4',",
+        "  SECRET 'rustfs_s3',",
+        f"  SIGV4_REGION {sql_string(region)},",
+        f"  SIGV4_SERVICE {sql_string(rest_signing_name)},",
+        "  ACCESS_DELEGATION_MODE 'none',",
+        "  STAGE_CREATE_TABLES false,",
+        "  SKIP_CREATE_TABLE_METADATA_UPDATES true,",
+        "  DISABLE_MULTI_TABLE_COMMIT true,",
+        "  REMOVE_FILES_ON_DELETE false,",
+        "  PURGE_REQUESTED false,",
+        "  SUPPORT_NESTED_NAMESPACES false",
+        ");",
+        f"SELECT COUNT(*) AS row_count FROM {table_identifier};",
+    ]
+    return "\n".join(statements) + "\n"
+
+
+def duckdb_command(*, sql_file: str = "/tmp/rustfs-s3tables-duckdb-read.sql") -> str:
+    return shell_join(["duckdb", "-c", f".read {sql_file}"])
 
 
 def snowflake_sql_template(*, endpoint: str, warehouse: str, rest_path: str, namespace: str, table: str) -> str:
@@ -623,11 +684,11 @@ def live_conformance_evidence(
                     OrderedDict(
                         [
                             ("client", "DuckDB Iceberg"),
-                            ("scenario", "iceberg-scan-current-metadata-location"),
+                            ("scenario", "rest-catalog-single-table-read-write-cross-engine-negative-boundaries"),
                             ("expected_status", "pass"),
                             ("expected_row_count", 2),
-                            ("claim_after_pass", "manual-live-read-verified"),
-                            ("write_claim_after_pass", "not-claimed"),
+                            ("claim_after_pass", "automated-rest-catalog-smoke"),
+                            ("write_claim_after_pass", "single-table-automated-smoke"),
                         ]
                     ),
                     OrderedDict(
@@ -653,9 +714,9 @@ def live_conformance_evidence(
             (
                 "promotion_rules",
                 [
-                    "Keep PyIceberg as the only automated claim unless the run is executed by CI or a repeatable operator job.",
+                    "Keep PyIceberg and DuckDB automated claims tied to their repeatable smoke entrypoints and recorded client versions.",
                     "Promote Spark only to manual-live-verified when the exact RustFS build, Spark version, Iceberg version, SQL output, and row_count are recorded.",
-                    "Do not promote Trino or DuckDB write compatibility from read probes; write compatibility remains not-claimed.",
+                    "Keep Trino write compatibility not-claimed after its read probe and do not broaden DuckDB beyond the automated single-table scenarios.",
                     "Do not promote Snowflake or vendor catalog interoperability from a generated template without a repeatable live run.",
                     "Treat manual-live failures as compatibility findings and keep the previous public claim boundary.",
                 ],
@@ -1425,6 +1486,18 @@ def live_conformance_harness(
         region=region,
         metadata_location=metadata_location,
     )
+    duckdb_rest_sql = duckdb_rest_catalog_sql(
+        endpoint=endpoint,
+        warehouse=warehouse,
+        access_key=access_key,
+        secret_key=secret_key,
+        region=region,
+        catalog_name=catalog_name,
+        namespace=namespace,
+        table=table,
+        rest_path=rest_path,
+        rest_signing_name=rest_signing_name,
+    )
     snowflake_sql = snowflake_sql_template(
         endpoint=endpoint,
         warehouse=warehouse,
@@ -1553,14 +1626,25 @@ def live_conformance_harness(
                     OrderedDict(
                         [
                             ("name", "DuckDB Iceberg"),
-                            ("status", "manual-live-read-probe"),
+                            ("status", "automated-smoke"),
                             ("version", duckdb_version),
                             ("metadata_location", metadata_location),
                             ("sql_file", "/tmp/rustfs-s3tables-duckdb-read.sql"),
                             ("sql", duckdb_sql),
                             ("command", duckdb_command()),
                             ("expected", "iceberg_scan returns row_count=2 when metadata_location points at the current Iceberg metadata JSON"),
-                            ("write_compatibility", "not-claimed"),
+                            ("rest_catalog_sql_file", "/tmp/rustfs-s3tables-duckdb-rest.sql"),
+                            ("rest_catalog_sql", duckdb_rest_sql),
+                            (
+                                "rest_catalog_command",
+                                duckdb_command(sql_file="/tmp/rustfs-s3tables-duckdb-rest.sql"),
+                            ),
+                            (
+                                "rest_catalog_expected",
+                                "generic Iceberg REST ATTACH returns row_count=2 for an existing RustFS table",
+                            ),
+                            ("rest_catalog_write_compatibility", "single-table-automated-smoke"),
+                            ("write_compatibility", "single-table-automated-smoke"),
                         ]
                     ),
                     OrderedDict(
@@ -1627,6 +1711,7 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
     parser.add_argument("--print-operations-guide", action="store_true")
     parser.add_argument("--print-spark-config", action="store_true")
     parser.add_argument("--print-spark-sql", action="store_true")
+    parser.add_argument("--print-duckdb-rest-sql", action="store_true")
     return parser.parse_args(argv)
 
 
@@ -1734,6 +1819,24 @@ def run(args: argparse.Namespace, output: StringIO | None = None) -> None:
         else:
             output.write(sql)
         printed = True
+    if args.print_duckdb_rest_sql:
+        sql = duckdb_rest_catalog_sql(
+            endpoint=args.endpoint,
+            warehouse=args.warehouse,
+            access_key=args.access_key,
+            secret_key=args.secret_key,
+            region=args.region,
+            catalog_name=args.catalog_name,
+            namespace=args.namespace,
+            table=args.table,
+            rest_path=args.rest_path or "/iceberg",
+            rest_signing_name=args.rest_signing_name or "s3",
+        )
+        if output is None:
+            print(sql, end="")
+        else:
+            output.write(sql)
+        printed = True
     if not printed:
         print_json({"engine_compatibility": engine_compatibility_matrix()}, output)
 

scripts/table-catalog/pyiceberg_smoke.py

@@ -167,9 +167,9 @@ CLIENT_MATRIX: list[dict[str, str]] = [
     },
     {
         "client": "DuckDB Iceberg",
-        "status": "manual-live-read-probe",
-        "coverage": "generated httpfs/iceberg SQL using an operator-supplied current metadata location; write/commit is not claimed",
-        "entrypoint": "scripts/table-catalog/engine_compatibility.py --print-live-conformance",
+        "status": "automated-smoke",
+        "coverage": "metadata-location read plus generic REST catalog single-table DDL, DML, schema evolution, snapshots, canonical and compatibility signing, negative boundaries, endpoint-disabled multi-table mode, concurrent writers, and PyIceberg cross-read",
+        "entrypoint": "scripts/table-catalog/duckdb_smoke.py",
     },
     {
         "client": "Databend",

scripts/table-catalog/test_duckdb_smoke.py

@@ -0,0 +1,220 @@
+#!/usr/bin/env python3
+"""Unit tests for the RustFS DuckDB REST Catalog smoke helper."""
+
+from __future__ import annotations
+
+import argparse
+import contextlib
+import io
+import json
+import tempfile
+import unittest
+from pathlib import Path
+from types import SimpleNamespace
+from unittest import mock
+
+import duckdb_smoke
+
+
+class DuckDBSmokeTest(unittest.TestCase):
+    def args(self) -> argparse.Namespace:
+        return argparse.Namespace(
+            endpoint="http://127.0.0.1:9000",
+            access_key="rustfsadmin",
+            secret_key="rustfsadmin",
+            region="us-east-1",
+            bucket="rustfs-duckdb-smoke",
+            namespace="duckdb_smoke",
+            table="events",
+            duckdb="duckdb",
+            duckdb_version="1.5.5",
+            timeout=60.0,
+            cleanup=True,
+            replace=False,
+            insecure=False,
+            live_evidence_output=None,
+            rustfs_build="rustfs-test",
+            git_sha="abc123",
+            catalog_backing="object",
+            operator="test-operator",
+            run_timestamp_utc="2026-08-27T00:00:00Z",
+        )
+
+    def test_parse_args_rejects_unsafe_identifiers(self) -> None:
+        with contextlib.redirect_stderr(io.StringIO()):
+            with self.assertRaises(SystemExit):
+                duckdb_smoke.parse_args(["--namespace", "bad-name"])
+
+    def test_duckdb_client_version_removes_v_prefix(self) -> None:
+        process = SimpleNamespace(returncode=0, stdout="v1.5.5\n", stderr="")
+        with mock.patch.object(duckdb_smoke.subprocess, "run", return_value=process):
+            self.assertEqual(duckdb_smoke.duckdb_client_version("duckdb", 10), "1.5.5")
+
+    def test_parse_duckdb_json_accepts_multiple_query_batches(self) -> None:
+        batches = duckdb_smoke.parse_duckdb_json(
+            '[{"row_count":2}]\n[{"id":20},\n{"id":40}]\n'
+        )
+
+        self.assertEqual(batches[0][0]["row_count"], 2)
+        self.assertEqual([row["id"] for row in batches[1]], [20, 40])
+
+    def test_run_duckdb_does_not_parse_json_for_failed_process(self) -> None:
+        process = SimpleNamespace(returncode=1, stdout="not-json", stderr="expected failure")
+        with mock.patch.object(duckdb_smoke.subprocess, "run", return_value=process):
+            execution = duckdb_smoke.run_duckdb("duckdb", "SELECT 1", 10)
+
+        self.assertEqual(execution.returncode, 1)
+        self.assertEqual(execution.batches, [])
+
+    def test_concurrent_retry_markers_do_not_accept_generic_commit_errors(self) -> None:
+        args = self.args()
+        generic_failure = duckdb_smoke.DuckDBExecution(1, [], "", "commit failed: permission denied")
+        with mock.patch.object(duckdb_smoke, "run_duckdb", return_value=generic_failure):
+            with self.assertRaisesRegex(RuntimeError, "permission denied"):
+                duckdb_smoke.run_concurrent_inserts("duckdb", args, "events_write")
+
+    def test_canonical_sql_covers_single_table_lifecycle(self) -> None:
+        sql = duckdb_smoke.canonical_positive_sql(
+            self.args(),
+            "events_seed",
+            "events_write",
+            "events_purge",
+            "events_drop",
+        )
+
+        self.assertIn("STAGE_CREATE_TABLES false", sql)
+        self.assertIn("SKIP_CREATE_TABLE_METADATA_UPDATES true", sql)
+        self.assertIn("CREATE TABLE", sql)
+        self.assertIn("INSERT INTO", sql)
+        self.assertIn("UPDATE", sql)
+        self.assertIn("DELETE FROM", sql)
+        self.assertIn("MERGE INTO", sql)
+        self.assertIn("ALTER TABLE", sql)
+        self.assertIn("iceberg_snapshots", sql)
+        self.assertNotIn("DROP TABLE IF EXISTS", sql)
+        self.assertIn('DROP TABLE "rustfs_duckdb"."duckdb_smoke"."events_drop"', sql)
+
+    def test_prepare_smoke_tables_refuses_existing_identifiers_without_replace(self) -> None:
+        catalog = mock.Mock()
+        catalog.table_exists.side_effect = lambda identifier: identifier[1] == "events_write"
+
+        with self.assertRaisesRegex(RuntimeError, "duckdb_smoke.events_write"):
+            duckdb_smoke.prepare_smoke_tables(
+                catalog,
+                "duckdb_smoke",
+                ["events_seed", "events_write"],
+                replace=False,
+            )
+
+        catalog.drop_table.assert_not_called()
+
+    def test_prepare_smoke_tables_replaces_only_existing_identifiers_when_requested(self) -> None:
+        catalog = mock.Mock()
+        catalog.table_exists.side_effect = lambda identifier: identifier[1] == "events_write"
+
+        duckdb_smoke.prepare_smoke_tables(
+            catalog,
+            "duckdb_smoke",
+            ["events_seed", "events_write"],
+            replace=True,
+        )
+
+        catalog.drop_table.assert_called_once_with(("duckdb_smoke", "events_write"))
+
+    def test_cleanup_preserves_a_preexisting_namespace(self) -> None:
+        catalog = mock.Mock()
+        catalog.table_exists.return_value = False
+
+        result = duckdb_smoke.cleanup_tables(
+            catalog,
+            "duckdb_smoke",
+            ["events_seed", "events_write"],
+            drop_namespace=False,
+        )
+
+        self.assertEqual(result, "dropped-tables-preserved-existing-namespace")
+        catalog.drop_namespace.assert_not_called()
+
+    def test_alias_sql_uses_s3tables_signing(self) -> None:
+        sql = duckdb_smoke.alias_sql(self.args(), "events_write")
+
+        self.assertIn("ENDPOINT 'http://127.0.0.1:9000/_iceberg'", sql)
+        self.assertIn("SIGV4_SERVICE 's3tables'", sql)
+        self.assertIn("INSERT INTO", sql)
+        self.assertIn("alias_final_row_count", sql)
+
+    def test_boundary_sql_records_required_compatibility_options(self) -> None:
+        args = self.args()
+        stage_sql = duckdb_smoke.negative_sql(
+            args,
+            kind="stage-create",
+            seed_table="events_seed",
+            write_table="events_write",
+            purge_table="events_purge",
+        )
+        stage_attach = stage_sql.split('DETACH "bootstrap_stage-create";', 1)[1]
+        self.assertNotIn("STAGE_CREATE_TABLES false", stage_attach)
+        self.assertIn("CREATE TABLE", stage_attach)
+
+        purge_sql = duckdb_smoke.negative_sql(
+            args,
+            kind="purge",
+            seed_table="events_seed",
+            write_table="events_write",
+            purge_table="events_purge",
+        )
+        self.assertIn("PURGE_REQUESTED true", purge_sql)
+        self.assertIn('DROP TABLE "purge_requested"."duckdb_smoke"."events_purge"', purge_sql)
+
+        v3_sql = duckdb_smoke.negative_sql(
+            args,
+            kind="format-v3",
+            seed_table="events_seed",
+            write_table="events_write",
+            purge_table="events_purge",
+        )
+        self.assertIn("'format-version' = '3'", v3_sql)
+
+        multi_sql = duckdb_smoke.multi_table_sql(
+            args,
+            seed_table="events_seed",
+            write_table="events_write",
+            purge_table="events_purge",
+        )
+        self.assertIn("DISABLE_MULTI_TABLE_COMMIT true", multi_sql)
+        self.assertIn("BEGIN TRANSACTION", multi_sql)
+        self.assertIn("'non-atomic'", multi_sql)
+
+    def test_pyiceberg_args_use_canonical_catalog(self) -> None:
+        args = duckdb_smoke.pyiceberg_args(self.args())
+
+        self.assertEqual(args.rest_path, "/iceberg")
+        self.assertEqual(args.rest_signing_name, "s3")
+        self.assertEqual(args.bucket, "rustfs-duckdb-smoke")
+
+    def test_live_evidence_records_automated_duckdb_claim(self) -> None:
+        args = self.args()
+        result = duckdb_smoke.DuckDBSmokeResult(
+            client_version="1.5.5",
+            metadata_location="s3://rustfs-duckdb-smoke/metadata/00001.json",
+            row_count=2,
+            cleanup_result="dropped-tables-and-namespace",
+            checks={"canonical_rest_catalog": "pass"},
+        )
+        with tempfile.TemporaryDirectory() as temp_dir:
+            output = Path(temp_dir) / "evidence.json"
+            args.live_evidence_output = str(output)
+            with mock.patch.object(duckdb_smoke.sys, "argv", ["duckdb_smoke.py", "--secret-key", "secret"]):
+                duckdb_smoke.write_live_evidence(args, result)
+            document = json.loads(output.read_text(encoding="utf-8"))
+
+        evidence = document["live_conformance_evidence"]
+        self.assertEqual(evidence["client_name"], "DuckDB Iceberg")
+        self.assertEqual(evidence["claim"], "automated-rest-catalog-smoke")
+        self.assertIn("--secret-key '<redacted>'", evidence["command"])
+        self.assertNotIn("--secret-key secret", evidence["command"])
+        self.assertEqual(document["validation"]["status"], "accepted")
+
+
+if __name__ == "__main__":
+    unittest.main()

scripts/table-catalog/test_engine_compatibility.py

@@ -41,6 +41,15 @@ class EngineCompatibilityTest(unittest.TestCase):
         self.assertEqual(trino["status"], "manual-live-read-probe")
         self.assertContainsScenario(trino, "catalog-load", "manual-live-probe")
 
+        duckdb = by_client["DuckDB Iceberg"]
+        self.assertEqual(duckdb["status"], "automated-smoke")
+        self.assertEqual(duckdb["entrypoint"], "scripts/table-catalog/duckdb_smoke.py")
+        self.assertContainsScenario(duckdb, "metadata-read", "automated")
+        self.assertContainsScenario(duckdb, "catalog-attach", "automated")
+        self.assertContainsScenario(duckdb, "write-table", "automated")
+        self.assertContainsScenario(duckdb, "unsupported-boundaries", "automated")
+        self.assertContainsScenario(duckdb, "multi-table-mode", "automated")
+
     def test_spark_config_uses_rustfs_rest_catalog_and_s3fileio(self) -> None:
         config = engine_compatibility.spark_catalog_config(
             endpoint="http://127.0.0.1:9000",
@@ -158,6 +167,70 @@ class EngineCompatibilityTest(unittest.TestCase):
                 table="orders",
             )
 
+    def test_duckdb_rest_catalog_sql_uses_rustfs_compatibility_options(self) -> None:
+        sql = engine_compatibility.duckdb_rest_catalog_sql(
+            endpoint="http://127.0.0.1:9000",
+            warehouse="rustfs-s3table-smoke",
+            access_key="rustfsadmin",
+            secret_key="rustfsadmin",
+            region="us-east-1",
+            catalog_name="rustfs",
+            namespace="smoke",
+            table="events",
+            rest_path="/iceberg",
+            rest_signing_name="s3",
+        )
+
+        self.assertIn("CREATE OR REPLACE SECRET rustfs_s3", sql)
+        self.assertIn("ENDPOINT '127.0.0.1:9000'", sql)
+        self.assertIn("SCOPE 's3://rustfs-s3table-smoke'", sql)
+        self.assertIn("ATTACH 'rustfs-s3table-smoke' AS \"rustfs\"", sql)
+        self.assertIn("ENDPOINT 'http://127.0.0.1:9000/iceberg'", sql)
+        self.assertIn("SIGV4_SERVICE 's3'", sql)
+        self.assertIn("STAGE_CREATE_TABLES false", sql)
+        self.assertIn("SKIP_CREATE_TABLE_METADATA_UPDATES true", sql)
+        self.assertIn("DISABLE_MULTI_TABLE_COMMIT true", sql)
+        self.assertIn("REMOVE_FILES_ON_DELETE false", sql)
+        self.assertIn("PURGE_REQUESTED false", sql)
+        self.assertIn('FROM "rustfs"."smoke"."events"', sql)
+        self.assertNotIn("ENDPOINT_TYPE", sql)
+
+    def test_duckdb_rest_catalog_sql_supports_s3tables_alias(self) -> None:
+        sql = engine_compatibility.duckdb_rest_catalog_sql(
+            endpoint="https://rustfs.example",
+            warehouse="analytics",
+            access_key="access'key",
+            secret_key="secret'key",
+            region="us-east-1",
+            catalog_name="rustfs_compat",
+            namespace="smoke",
+            table="events",
+            rest_path="/_iceberg",
+            rest_signing_name="s3tables",
+        )
+
+        self.assertIn("USE_SSL true", sql)
+        self.assertIn("ENDPOINT 'rustfs.example'", sql)
+        self.assertIn("ENDPOINT 'https://rustfs.example/_iceberg'", sql)
+        self.assertIn("SIGV4_SERVICE 's3tables'", sql)
+        self.assertIn("KEY_ID 'access''key'", sql)
+        self.assertIn("SECRET 'secret''key'", sql)
+
+    def test_duckdb_rest_catalog_sql_rejects_endpoint_without_scheme(self) -> None:
+        with self.assertRaisesRegex(ValueError, "must include http:// or https://"):
+            engine_compatibility.duckdb_rest_catalog_sql(
+                endpoint="127.0.0.1:9000",
+                warehouse="analytics",
+                access_key="rustfsadmin",
+                secret_key="rustfsadmin",
+                region="us-east-1",
+                catalog_name="rustfs",
+                namespace="smoke",
+                table="events",
+                rest_path="/iceberg",
+                rest_signing_name="s3",
+            )
+
     def test_cli_prints_machine_readable_engine_matrix(self) -> None:
         payload = engine_compatibility.cli_json(["--print-engine-matrix"])
         document = json.loads(payload)
@@ -308,6 +381,27 @@ class EngineCompatibilityTest(unittest.TestCase):
         self.assertEqual(config["spark.sql.catalog.rustfs.uri"], "http://127.0.0.1:9000/_iceberg")
         self.assertEqual(config["spark.sql.catalog.rustfs.rest.signing-name"], "s3tables")
 
+    def test_cli_prints_duckdb_rest_catalog_sql(self) -> None:
+        sql = engine_compatibility.cli_json(
+            [
+                "--print-duckdb-rest-sql",
+                "--endpoint",
+                "http://127.0.0.1:9000",
+                "--warehouse",
+                "analytics",
+                "--catalog-name",
+                "rustfs_compat",
+                "--rest-path",
+                "/_iceberg",
+                "--rest-signing-name",
+                "s3tables",
+            ]
+        )
+
+        self.assertIn("ATTACH 'analytics' AS \"rustfs_compat\"", sql)
+        self.assertIn("ENDPOINT 'http://127.0.0.1:9000/_iceberg'", sql)
+        self.assertIn("SIGV4_SERVICE 's3tables'", sql)
+
     def test_live_conformance_harness_pins_clients_and_records_commands(self) -> None:
         harness = engine_compatibility.live_conformance_harness(
             endpoint="http://127.0.0.1:9000",
@@ -359,11 +453,16 @@ class EngineCompatibilityTest(unittest.TestCase):
         self.assertEqual(trino["write_compatibility"], "not-claimed")
 
         duckdb = by_client["DuckDB Iceberg"]
-        self.assertEqual(duckdb["status"], "manual-live-read-probe")
+        self.assertEqual(duckdb["status"], "automated-smoke")
+        self.assertEqual(duckdb["version"], "1.5.5")
         self.assertIn("LOAD httpfs", duckdb["sql"])
         self.assertIn("LOAD iceberg", duckdb["sql"])
         self.assertIn("iceberg_scan", duckdb["sql"])
-        self.assertEqual(duckdb["write_compatibility"], "not-claimed")
+        self.assertIn("ATTACH 'rustfs-s3table-smoke'", duckdb["rest_catalog_sql"])
+        self.assertIn("STAGE_CREATE_TABLES false", duckdb["rest_catalog_sql"])
+        self.assertIn("SKIP_CREATE_TABLE_METADATA_UPDATES true", duckdb["rest_catalog_sql"])
+        self.assertEqual(duckdb["rest_catalog_write_compatibility"], "single-table-automated-smoke")
+        self.assertEqual(duckdb["write_compatibility"], "single-table-automated-smoke")
 
         snowflake = by_client["Snowflake Open Catalog / Iceberg integrations"]
         self.assertEqual(snowflake["status"], "manual-reference-probe")
@@ -407,7 +506,8 @@ class EngineCompatibilityTest(unittest.TestCase):
         self.assertEqual(table_by_client["PyIceberg"]["claim_after_pass"], "automated-smoke")
         self.assertEqual(table_by_client["Spark Iceberg REST catalog"]["claim_after_pass"], "manual-live-verified")
         self.assertEqual(table_by_client["Trino Iceberg REST catalog"]["write_claim_after_pass"], "not-claimed")
-        self.assertEqual(table_by_client["DuckDB Iceberg"]["write_claim_after_pass"], "not-claimed")
+        self.assertEqual(table_by_client["DuckDB Iceberg"]["claim_after_pass"], "automated-rest-catalog-smoke")
+        self.assertEqual(table_by_client["DuckDB Iceberg"]["write_claim_after_pass"], "single-table-automated-smoke")
         self.assertIn("manual-live", " ".join(evidence["promotion_rules"]))
         self.assertIn("not-claimed", " ".join(evidence["promotion_rules"]))
 
@@ -496,6 +596,10 @@ class EngineCompatibilityTest(unittest.TestCase):
         self.assertIn("metadata_location", schema["required_fields"])
         self.assertIn("claim", schema["required_fields"])
         self.assertEqual(schema["claim_promotion"]["Trino Iceberg REST catalog"], ["manual-live-read-verified"])
+        self.assertEqual(
+            schema["claim_promotion"]["DuckDB Iceberg"],
+            ["manual-live-read-verified", "automated-rest-catalog-smoke"],
+        )
 
     def test_production_operations_guide_covers_release_boundaries(self) -> None:
         guide = engine_compatibility.production_operations_guide(

scripts/test/rustfs_heal_test.md

@@ -0,0 +1,106 @@
+# RustFS Heal Test
+
+Node-outage heal test driven by
+[`scripts/test/rustfs_heal_test.sh`](rustfs_heal_test.sh), based on the
+Obsidian note "RustFS Heal 测试步骤". Uses the same 3-node test environment as
+the pool expansion test (`vm000 vm001 vm002`).
+
+All status checks talk to the RustFS admin API directly (SigV4-signed,
+`jq` assertions), no `rc` required.
+
+## What it does
+
+1. Downloads the `.deb` package on all nodes (release tag or a direct URL such
+   as the nightly/R2 package).
+2. Installs it, writes the 3x4 config
+   (`http://rustfs-node{1...3}:9000/data/rustfs{1...4}/mnmd`), starts all
+   three nodes simultaneously, verifies the cluster is up.
+3. Writes data with `warp` while monitoring disk usage on the surviving nodes
+   (`df -B1G | grep /data/rustfs`):
+   - when both surviving nodes reach `STOP_NODE_AT_GB` (default 15 GiB), stop
+     the outage node (`vm002`, `OUTAGE_NODE_INDEX=2`);
+   - keep writing until both surviving nodes reach `WARP_STOP_AT_GB`
+     (default 40 GiB), then stop warp.
+4. Restarts the outage node.
+5. Starts cluster heal: `POST /rustfs/admin/v3/heal/` with body
+   `{"recursive":true}` (retried, returns a `clientToken`).
+6. Monitors the heal task via `POST /rustfs/admin/v3/heal/?clientToken=<token>`
+   until the server verdict is a terminal success (`finished`/`completed`) with
+   `objects_failed == 0`.
+7. Result analysis: heal stats (scanned/healed/failed), an **S3 read-back
+   verification** of the written objects (list the test bucket and GET a
+   sample — every read must succeed), per-node disk usage (observability),
+   pass/fail verdict.
+
+Success is the server's own scan/repair verdict (heal finished, 0 failed)
+**plus** an end-to-end data read-back; per-node disk usage is logged as
+observability, not a pass gate (EC distributes different shards per node, so a
+fixed per-node GB target is not a meaningful invariant).
+
+## Self-hosted runner prerequisites
+
+- Register the admin host (e.g. `heal`) as a runner with the
+  `smoke-testing` label.
+- Install `jq`, `openssl`, `curl` and `warp` on the runner. `rc` is **not**
+  required.
+- The runner user must be able to SSH to `vm000/vm001/vm002` without a
+  password prompt; nodes need passwordless `sudo` for the SSH user and
+  resolvable `rustfs-node*` hostnames.
+- Admin API credentials need the `admin:server-info`, `admin:heal` and
+  `admin:rebalance` actions.
+
+## Configuration
+
+Same repository secrets/variables as the pool expansion workflow:
+
+| Kind   | Name                  | Purpose                                        |
+| ------ | --------------------- | ---------------------------------------------- |
+| Secret | `RUSTFS_ACCESS_KEY`   | RustFS access key (default `rustfs@test`)      |
+| Secret | `RUSTFS_SECRET_KEY`   | RustFS secret key (default `rustfs@test`)      |
+| Var    | `RUSTFS_API_ENDPOINT` | Admin API endpoint, e.g. `http://127.0.0.1:9000` (`RUSTFS_RC_ENDPOINT` fallback) |
+| Var    | `RUSTFS_NODES`        | `vm000 vm001 vm002`                            |
+| Var    | `RUSTFS_SSH_USER`     | `azureuser`                                    |
+| Var    | `RUSTFS_NIGHTLY_PACKAGE_URL` | Default nightly deb URL (defaults to the R2 `latest` alias) |
+
+## Workflow inputs
+
+| Input            | Default | Meaning                                   |
+| ---------------- | ------- | ----------------------------------------- |
+| `package_url`    | nightly | Direct `.deb` URL; empty = latest nightly |
+| `stop_node_gb`   | `15`    | Stop outage node at N GiB on survivors    |
+| `warp_stop_gb`   | `40`    | Stop warp at N GiB on survivors           |
+| `cleanup_before` | `true`  | Reset nodes before the test               |
+| `cleanup_after`  | `true`  | Reset nodes after the test                |
+
+> ⚠️ `--reset` purges the `rustfs` package and deletes the data directories on
+> all nodes. Only run against a dedicated test environment.
+
+## Manual usage
+
+```bash
+./scripts/test/rustfs_heal_test.sh --all -y \
+  --package-url https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb \
+  --endpoint http://127.0.0.1:9000
+
+./scripts/test/rustfs_heal_test.sh --steps 5,6,7
+./scripts/test/rustfs_heal_test.sh --reset -y
+```
+
+## Known issues
+
+- Nightly builds gate pool/rebalance activation on a live fleet capability
+  proof (rustfs/backlog#2031); the script retries heal/rebalance starts and
+  prints a hint when the signature appears.
+- The cluster-level `GET /rustfs/admin/v3/background-heal/status` aggregator
+  returns 501 in the single-pool 3x4 topology (no notification system), so the
+  script monitors the started heal task via its `clientToken` instead.
+- The heal task may report `progress: null` while running; the script logs
+  this as evidence (rustfs/backlog#2035) rather than coercing it to zero, and
+  reads the canonical camelCase progress fields
+  (`objectsScanned`/`objectsHealed`/`objectsFailed`/`progressPercentage`) with
+  a snake_case fallback.
+- The server-side per-task heal timeout defaults to 5 minutes; the script
+  writes `RUSTFS_HEAL_TASK_TIMEOUT_SECS=21600` (6h) into the node config so a
+  multi-tens-of-GiB heal can finish. The background scanner is disabled
+  (`RUSTFS_HEAL_AUTO_HEAL_ENABLE=false`) so the explicit heal is the only
+  repair mechanism and the outage effect stays observable.

scripts/test/rustfs_performance_test.sh

@@ -0,0 +1,543 @@
+#!/usr/bin/env bash
+#
+# rustfs-performance-test.sh
+# RustFS 4x4 集群性能压测全流程脚本
+#
+# Based on the Obsidian note "RustFS 性能测试". Full workflow:
+#   1. Cleanup: stop & purge rustfs, remove data dirs on all nodes
+#   2. Download the RustFS package on all nodes
+#   3. Install RustFS on all nodes (dpkg -i), recreate volume dirs
+#   4. Write /etc/default/rustfs (4-node x 4-drive MNMD), start all nodes
+#      in parallel and verify the service is Running
+#   5. Run the benchmark (warp GET/PUT/MIXED via rustfs-performance-testing.sh)
+#   6. Analyze results (summary.tsv / summary.md)
+#   7. Final cleanup: stop & purge rustfs, remove data dirs
+#
+# The script is driven from an admin host (e.g. a jumpbox) and operates on
+# the target nodes over SSH, mirroring scripts/test/rustfs_*_test.sh.
+#
+# Usage:
+#   ./rustfs-performance-test.sh --all                  # run all steps 1-7
+#   ./rustfs-performance-test.sh --step 5               # run a single step
+#   ./rustfs-performance-test.sh --steps 2,3,4          # run selected steps
+#   ./rustfs-performance-test.sh --all --dry-run        # preview only
+#   ./rustfs-performance-test.sh --all -y --package-url <deb URL>
+#
+# Notes:
+#   - SSH user defaults to azureuser (passwordless sudo on the nodes);
+#     pass --ssh-user root if your nodes accept root login.
+#   - The benchmark runner defaults to
+#     ~/Documents/Obsidian Vault/rustfs-performance-testing.sh; override with
+#     --bench-script / RUSTFS_BENCH_SCRIPT. warp must be installed on the
+#     admin host.
+#   - Steps 1 and 7 destroy the RustFS install and all data (confirmed).
+#
+set -Eeuo pipefail
+
+# ==================== Configuration (adjust to your environment) ====================
+
+# Target nodes (4x4: 4 nodes x 4 drives each)
+if [ -n "${RUSTFS_NODES:-}" ]; then
+  read -r -a NODES <<<"${RUSTFS_NODES}"
+else
+  NODES=(vm000 vm001 vm002 vm003)
+fi
+
+SSH_USER="${RUSTFS_SSH_USER:-azureuser}"
+SSH_PORT="${RUSTFS_SSH_PORT:-22}"
+SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new -p "${SSH_PORT}")
+
+# Package: GitHub release tag, e.g. "1.0.0-rc.3". PACKAGE_URL is derived from
+# RUSTFS_VERSION unless --package-url / RUSTFS_PACKAGE_URL is given.
+RUSTFS_VERSION="${RUSTFS_VERSION:-1.0.0-rc.3}"
+PACKAGE_URL="${RUSTFS_PACKAGE_URL:-}"
+ARCH="${RUSTFS_ARCH:-amd64}"
+PACKAGES_DIR="/home/rustfs/packages"
+PACKAGE_FILE="rustfs.deb"
+PACKAGE_SHA256="${RUSTFS_PACKAGE_SHA256:-}"
+
+# 4x4 topology: 4 nodes x 4 drives each, same expression on every node
+DRIVES_PER_NODE="${RUSTFS_DRIVES_PER_NODE:-4}"
+VOLUMES="http://rustfs-node{1...4}:9000/data/rustfs{1...4}/mnmd"
+
+# RustFS service configuration (written to /etc/default/rustfs)
+RUSTFS_CONFIG_FILE="/etc/default/rustfs"
+RUSTFS_SERVICE="rustfs"
+RUSTFS_PACKAGE_NAME="rustfs"
+RUSTFS_USER="rustfs"
+ACCESS_KEY="${RUSTFS_ACCESS_KEY:-rustfs@test}"
+SECRET_KEY="${RUSTFS_SECRET_KEY:-rustfs@test}"
+RUSTFS_ADDRESS=":9000"
+RUSTFS_CONSOLE_ADDRESS=":9001"
+RUSTFS_CONSOLE_ENABLE=true
+RUSTFS_OBS_LOGGER_LEVEL=error
+RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
+
+# Benchmark runner (step 5/6): prefer the default Obsidian location, fall back
+# to a rustfs-performance-testing.sh next to this script (e.g. in the repo or
+# on a jumpbox).
+_DEFAULT_BENCH="${HOME}/Documents/Obsidian Vault/rustfs-performance-testing.sh"
+_SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+if [ -x "${_DEFAULT_BENCH}" ]; then
+  _BENCH_RESOLVED="${_DEFAULT_BENCH}"
+elif [ -x "${_SCRIPT_DIR}/rustfs-performance-testing.sh" ]; then
+  _BENCH_RESOLVED="${_SCRIPT_DIR}/rustfs-performance-testing.sh"
+elif [ -x "${_SCRIPT_DIR}/rustfs_performance_testing.sh" ]; then
+  _BENCH_RESOLVED="${_SCRIPT_DIR}/rustfs_performance_testing.sh"
+else
+  _BENCH_RESOLVED="${_DEFAULT_BENCH}"
+fi
+BENCH_SCRIPT="${RUSTFS_BENCH_SCRIPT:-${_BENCH_RESOLVED}}"
+RESULT_DIR="${RUSTFS_RESULT_DIR:-$(pwd)/warp-bench-results-$(date +%Y%m%d-%H%M%S)}"
+WARP_HOST="${RUSTFS_WARP_HOST:-rustfs-node1:9000,rustfs-node2:9000,rustfs-node3:9000,rustfs-node4:9000}"
+WARP_BUCKET="${RUSTFS_WARP_BUCKET:-warp-benchmark-bucket}"
+WARP_CONCURRENCY="${RUSTFS_WARP_CONCURRENCY:-64}"
+WARP_DURATION="${RUSTFS_WARP_DURATION:-5m}"
+WARP_GET_OBJECTS="${RUSTFS_WARP_GET_OBJECTS:-2500}"
+WARP_SLEEP="${RUSTFS_WARP_SLEEP:-60}"
+# Manual method/size selection (passed through to the benchmark runner; empty = full run)
+WARP_METHODS="${RUSTFS_WARP_METHODS:-}"
+WARP_SIZES="${RUSTFS_WARP_SIZES:-}"
+
+# Timeouts (seconds)
+SERVICE_TIMEOUT="${RUSTFS_SERVICE_TIMEOUT:-300}"
+POLL_INTERVAL="${RUSTFS_POLL_INTERVAL:-10}"
+
+# ==================== Runtime options (set by CLI) ====================
+DRY_RUN=0
+ASSUME_YES=0
+SKIP_DOWNLOAD=0
+PREFLIGHT=0
+LOG_FILE=""
+SELECTED_STEPS=()
+
+# ==================== Helpers ====================
+
+log()  { printf '\033[1;36m[INFO]\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33m[WARN]\033[0m %s\n' "$*"; }
+die()  { printf '\033[1;31m[ERROR]\033[0m %s\n' "$*" >&2; exit 1; }
+
+confirm() {
+  if [ "${ASSUME_YES}" -eq 1 ] || [ "${DRY_RUN}" -eq 1 ]; then return 0; fi
+  printf '\033[1;33m[CONFIRM]\033[0m %s (y/N) ' "$1"
+  read -r answer
+  case "${answer}" in
+    y|Y|yes|YES) return 0 ;;
+    *) die "cancelled" ;;
+  esac
+}
+
+need_cmd() {
+  [ "${DRY_RUN}" -eq 1 ] && return 0
+  command -v "$1" >/dev/null 2>&1 || die "missing command: $1 ($2); install it first"
+}
+
+# Run a remote script on a single node (script is read from stdin)
+run_remote() {
+  local node="$1" script
+  script="$(cat)"
+  if [ "${DRY_RUN}" -eq 1 ]; then
+    log "DRY-RUN: ssh ${SSH_USER}@${node} <<'REMOTE'"
+    printf '%s\n' "${script}" | sed 's/^/    | /'
+    log "DRY-RUN: ----"
+    return 0
+  fi
+  log "==> ${node}: executing remote script"
+  ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" 'bash -s' <<<"${script}"
+}
+
+# Run the same remote script on all nodes in parallel (script from stdin)
+run_remote_all() {
+  local script pids=() i=0 fail=0
+  script="$(cat)"
+  for node in "${NODES[@]}"; do
+    if [ "${DRY_RUN}" -eq 1 ]; then
+      log "DRY-RUN: ssh ${SSH_USER}@${node} <<'REMOTE'"
+      printf '%s\n' "${script}" | sed 's/^/    | /'
+      log "DRY-RUN: ----"
+    else
+      log "==> ${node}: executing remote script"
+      ( ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" 'bash -s' <<<"${script}" ) &
+      pids[$i]=$!
+      i=$((i+1))
+    fi
+  done
+  if [ "${#pids[@]}" -gt 0 ]; then
+    for pid in "${pids[@]}"; do
+      wait "${pid}" || fail=1
+    done
+  fi
+  [ "${fail}" -eq 0 ] || die "one or more remote executions failed"
+}
+
+rustfs_config_body() {
+  cat <<EOF
+RUSTFS_ACCESS_KEY=${ACCESS_KEY}
+RUSTFS_SECRET_KEY=${SECRET_KEY}
+RUSTFS_VOLUMES="${VOLUMES}"
+RUSTFS_ADDRESS="${RUSTFS_ADDRESS}"
+RUSTFS_CONSOLE_ADDRESS="${RUSTFS_CONSOLE_ADDRESS}"
+RUSTFS_CONSOLE_ENABLE=${RUSTFS_CONSOLE_ENABLE}
+RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL}
+RUSTFS_OBS_LOG_DIRECTORY="${RUSTFS_OBS_LOG_DIRECTORY}"
+EOF
+}
+
+write_rustfs_config() {
+  local node="$1" body
+  body="$(rustfs_config_body)"
+  log "${node}: writing config ${RUSTFS_CONFIG_FILE}"
+  {
+    printf 'set -euo pipefail\n'
+    printf 'SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo -n"\n'
+    printf '%s tee %s >/dev/null <<RUSTFS_EOF\n' '${SUDO}' "${RUSTFS_CONFIG_FILE}"
+    printf '%s' "${body}"
+    printf '\nRUSTFS_EOF\n'
+    printf '${SUDO} systemctl daemon-reload\n'
+  } | run_remote "${node}"
+}
+
+service_action() {
+  local action="$1" node="$2"
+  log "${node}: systemctl ${action} ${RUSTFS_SERVICE}"
+  [ "${DRY_RUN}" -eq 1 ] && return 0
+  ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" \
+    "if [ \"\$(id -u)\" -ne 0 ]; then sudo -n systemctl ${action} ${RUSTFS_SERVICE}; else systemctl ${action} ${RUSTFS_SERVICE}; fi" \
+    || die "${node}: systemctl ${action} failed"
+}
+
+wait_service_active() {
+  local node="$1" elapsed=0
+  log "${node}: waiting for ${RUSTFS_SERVICE} to become active"
+  [ "${DRY_RUN}" -eq 1 ] && { log "${node}: (dry-run) skip wait"; return 0; }
+  while [ "${elapsed}" -lt "${SERVICE_TIMEOUT}" ]; do
+    if ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" \
+        "systemctl is-active ${RUSTFS_SERVICE} 2>/dev/null" | grep -q active; then
+      log "${node}: service active"
+      return 0
+    fi
+    sleep "${POLL_INTERVAL}"
+    elapsed=$((elapsed + POLL_INTERVAL))
+  done
+  die "${node}: ${RUSTFS_SERVICE} did not become active within ${SERVICE_TIMEOUT}s"
+}
+
+verify_service_running() {
+  local node="$1"
+  log "${node}: checking service status"
+  if [ "${DRY_RUN}" -eq 0 ]; then
+    ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" \
+      "systemctl status ${RUSTFS_SERVICE} --no-pager | head -n 12" || true
+  fi
+}
+
+build_package_url() {
+  local asset
+  asset="rustfs_$(printf '%s' "${RUSTFS_VERSION}" | tr '-' '.')_${ARCH}.deb"
+  printf 'https://github.com/rustfs/rustfs/releases/download/%s/%s' "${RUSTFS_VERSION}" "${asset}"
+}
+
+resolve_package_url() {
+  if [ -n "${PACKAGE_URL}" ]; then printf '%s' "${PACKAGE_URL}"; else build_package_url; fi
+}
+
+preflight() {
+  log "preflight checks"
+  need_cmd ssh "openssh client"
+  need_cmd curl "http client"
+  need_cmd warp "warp benchmark tool (for step 5)"
+  if [ ! -x "${BENCH_SCRIPT}" ]; then
+    die "benchmark script not found or not executable: ${BENCH_SCRIPT}"
+  fi
+  if [ "${DRY_RUN}" -eq 0 ]; then
+    for node in "${NODES[@]}"; do
+      ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" 'echo ok' >/dev/null \
+        || die "cannot ssh to ${node}"
+    done
+    log "all nodes reachable: ${NODES[*]}"
+  fi
+  log "preflight OK"
+}
+
+# ==================== Steps ====================
+
+step1_cleanup() {
+  log "step 1: cleanup environment on all nodes (stop & purge rustfs, remove data dirs)"
+  confirm "This DESTROYS the RustFS install and ALL data on ${NODES[*]} (irreversible). Continue?"
+  local script
+  script="$(cat <<EOF
+set -euo pipefail
+SUDO=""; [ "\$(id -u)" -ne 0 ] && SUDO="sudo -n"
+\${SUDO} systemctl stop rustfs 2>/dev/null || true
+if \${SUDO} dpkg -l rustfs 2>/dev/null | grep -q "^ii"; then
+  \${SUDO} dpkg -P rustfs
+  echo "purged rustfs"
+else
+  echo "rustfs not installed, skip purge"
+fi
+for i in \$(seq 1 ${DRIVES_PER_NODE}); do
+  \${SUDO} rm -rf /data/rustfs\${i}/mnmd
+  \${SUDO} mkdir -p /data/rustfs\${i}/mnmd
+  \${SUDO} chown -R rustfs:rustfs /data/rustfs\${i}/mnmd
+done
+echo "cleanup done on \$(hostname)"
+EOF
+)"
+  printf '%s\n' "${script}" | run_remote_all
+  log "step 1 complete"
+}
+
+step2_download() {
+  log "step 2: download the package on all nodes"
+  local url script
+  url="$(resolve_package_url)"
+  script="$(cat <<EOF
+set -euo pipefail
+SUDO=""; [ "\$(id -u)" -ne 0 ] && SUDO="sudo -n"
+if [ -f "${PACKAGES_DIR}/${PACKAGE_FILE}" ] && [ "${SKIP_DOWNLOAD}" -eq 1 ]; then
+  echo "already exists: ${PACKAGES_DIR}/${PACKAGE_FILE}, skipping download"
+else
+  echo "downloading ${url} ..."
+  curl -fSL --retry 3 -o "/tmp/${PACKAGE_FILE}" "${url}"
+  \${SUDO} mkdir -p "${PACKAGES_DIR}"
+  \${SUDO} install -m 0644 "/tmp/${PACKAGE_FILE}" "${PACKAGES_DIR}/${PACKAGE_FILE}"
+  \${SUDO} rm -f "/tmp/${PACKAGE_FILE}"
+fi
+if [ -n "${PACKAGE_SHA256}" ]; then
+  echo "${PACKAGE_SHA256}  ${PACKAGES_DIR}/${PACKAGE_FILE}" | sha256sum -c - || { echo "checksum verification failed"; exit 1; }
+fi
+ls -lh "${PACKAGES_DIR}/${PACKAGE_FILE}"
+EOF
+)"
+  printf '%s\n' "${script}" | run_remote_all
+  log "step 2 complete"
+}
+
+step3_install() {
+  log "step 3: install the RustFS service on all nodes"
+  confirm "About to run dpkg -i ${PACKAGE_FILE} on all nodes. Continue?"
+  local script
+  script="$(cat <<'EOF'
+set -euo pipefail
+SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo -n"
+${SUDO} dpkg -i /home/rustfs/packages/rustfs.deb
+${SUDO} systemctl daemon-reload
+echo "--- installed package ---"
+dpkg -l rustfs | tail -n 1
+EOF
+)"
+  printf '%s\n' "${script}" | run_remote_all
+  log "step 3 complete"
+}
+
+step4_configure_start() {
+  log "step 4: write config, start and verify the service on all nodes"
+  local node
+  for node in "${NODES[@]}"; do
+    write_rustfs_config "${node}"
+  done
+  for node in "${NODES[@]}"; do
+    service_action start "${node}" &
+  done
+  wait
+  for node in "${NODES[@]}"; do
+    wait_service_active "${node}"
+    verify_service_running "${node}"
+  done
+  log "step 4 complete"
+}
+
+step5_benchmark() {
+  log "step 5: run the benchmark (${BENCH_SCRIPT})"
+  need_cmd warp "warp benchmark tool"
+  confirm "About to run the full GET/PUT/MIXED benchmark (~6-10 hours). Continue?"
+  if [ "${DRY_RUN}" -eq 1 ]; then
+    log "DRY-RUN: WARP_HOST=${WARP_HOST} WARP_RESULT_DIR=${RESULT_DIR} bash ${BENCH_SCRIPT}"
+    return 0
+  fi
+  WARP_HOST="${WARP_HOST}" \
+  WARP_ACCESS_KEY="${ACCESS_KEY}" \
+  WARP_SECRET_KEY="${SECRET_KEY}" \
+  WARP_BUCKET="${WARP_BUCKET}" \
+  WARP_CONCURRENCY="${WARP_CONCURRENCY}" \
+  WARP_DURATION="${WARP_DURATION}" \
+  WARP_GET_OBJECTS="${WARP_GET_OBJECTS}" \
+  WARP_SLEEP_BETWEEN_ROUNDS="${WARP_SLEEP}" \
+  WARP_METHODS="${WARP_METHODS}" \
+  WARP_SIZES="${WARP_SIZES}" \
+  WARP_RESULT_DIR="${RESULT_DIR}" \
+  bash "${BENCH_SCRIPT}"
+  log "step 5 complete (results in ${RESULT_DIR})"
+}
+
+step6_analyze() {
+  log "step 6: analyze results from ${RESULT_DIR}"
+  if [ "${DRY_RUN}" -eq 1 ]; then
+    log "DRY-RUN: bash ${BENCH_SCRIPT} --parse-only ${RESULT_DIR}"
+    return 0
+  fi
+  if [ ! -d "${RESULT_DIR}" ]; then
+    die "result directory not found: ${RESULT_DIR}"
+  fi
+  if [ -f "${RESULT_DIR}/summary.md" ]; then
+    log "summary already generated: ${RESULT_DIR}/summary.md"
+  else
+    log "generating summary with --parse-only"
+    bash "${BENCH_SCRIPT}" --parse-only "${RESULT_DIR}"
+  fi
+  log "----- summary.md -----"
+  cat "${RESULT_DIR}/summary.md"
+  log "step 6 complete"
+}
+
+step7_cleanup() {
+  log "step 7: final cleanup on all nodes (stop & purge rustfs, remove data dirs)"
+  confirm "This DESTROYS the RustFS install and ALL data on ${NODES[*]} (irreversible). Continue?"
+  local script
+  script="$(cat <<EOF
+set -euo pipefail
+SUDO=""; [ "\$(id -u)" -ne 0 ] && SUDO="sudo -n"
+\${SUDO} systemctl stop rustfs 2>/dev/null || true
+if \${SUDO} dpkg -l rustfs 2>/dev/null | grep -q "^ii"; then
+  \${SUDO} dpkg -P rustfs
+  echo "purged rustfs"
+fi
+for i in \$(seq 1 ${DRIVES_PER_NODE}); do
+  \${SUDO} rm -rf /data/rustfs\${i}/mnmd
+done
+echo "cleanup done on \$(hostname)"
+EOF
+)"
+  printf '%s\n' "${script}" | run_remote_all
+  log "step 7 complete"
+}
+
+# ==================== CLI ====================
+
+usage() {
+  cat <<'USAGE'
+Usage: ./rustfs-performance-test.sh [options]
+
+Steps:
+  1  cleanup environment (purge rustfs, remove data dirs)   [destructive]
+  2  download the RustFS package on all nodes
+  3  install RustFS (dpkg -i)
+  4  write config, start service, verify Running
+  5  run benchmark (warp GET/PUT/MIXED)
+  6  analyze results (summary.tsv / summary.md)
+  7  final cleanup (purge rustfs, remove data dirs)         [destructive]
+
+Options:
+  --all                     Run all steps 1-7
+  --step N                  Run a single step
+  --steps 1,3,5-7           Run selected steps
+  --version VERSION         GitHub release tag (default 1.0.0-rc.3)
+  --package-url URL         Direct deb URL (overrides --version)
+  --sha256 HASH             Verify package checksum
+  --skip-download           Keep an existing package file
+  --bench-script PATH       Benchmark runner (default: Obsidian Vault rustfs-performance-testing.sh)
+  --result-dir DIR          Benchmark result directory
+  --warp-duration DUR       warp duration per round (default 5m)
+  --warp-concurrency N      warp concurrency (default 64)
+  --ssh-user USER           SSH user (default azureuser)
+  --ssh-port PORT           SSH port (default 22)
+  --preflight               Check environment and exit
+  --log-file FILE           Append all output to FILE
+  --dry-run                 Preview commands without executing them
+  -y, --yes                 Skip all confirmation prompts
+  -h, --help                Show this help
+
+Examples:
+  ./rustfs-performance-test.sh --all
+  ./rustfs-performance-test.sh --all --dry-run
+  ./rustfs-performance-test.sh --all -y --package-url https://dl.rustfs.com/...deb
+  ./rustfs-performance-test.sh --step 5
+USAGE
+}
+
+expand_steps() {
+  local spec="$1" part start end i
+  IFS=',' read -ra parts <<<"${spec}"
+  for part in "${parts[@]}"; do
+    if [[ "${part}" =~ ^([0-9]+)-([0-9]+)$ ]]; then
+      start="${BASH_REMATCH[1]}"; end="${BASH_REMATCH[2]}"
+      for ((i=start; i<=end; i++)); do SELECTED_STEPS+=("${i}"); done
+    elif [[ "${part}" =~ ^[0-9]+$ ]]; then
+      SELECTED_STEPS+=("${part}")
+    else
+      die "cannot parse step spec: ${part}"
+    fi
+  done
+}
+
+run_steps() {
+  local step
+  for step in "${SELECTED_STEPS[@]}"; do
+    case "${step}" in
+      1) step1_cleanup ;;
+      2) step2_download ;;
+      3) step3_install ;;
+      4) step4_configure_start ;;
+      5) step5_benchmark ;;
+      6) step6_analyze ;;
+      7) step7_cleanup ;;
+      *) die "unknown step: ${step}" ;;
+    esac
+    log "step ${step} completed"
+  done
+}
+
+main() {
+  [ "$#" -eq 0 ] && { usage; exit 0; }
+  local opt all=0
+  while [ "$#" -gt 0 ]; do
+    opt="$1"; shift
+    case "${opt}" in
+      --all) all=1 ;;
+      --step) SELECTED_STEPS+=("$1"); shift ;;
+      --steps) expand_steps "$1"; shift ;;
+      --version) RUSTFS_VERSION="$1"; shift ;;
+      --package-url) PACKAGE_URL="$1"; shift ;;
+      --sha256) PACKAGE_SHA256="$1"; shift ;;
+      --skip-download) SKIP_DOWNLOAD=1 ;;
+      --bench-script) BENCH_SCRIPT="$1"; shift ;;
+      --result-dir) RESULT_DIR="$1"; shift ;;
+      --warp-duration) WARP_DURATION="$1"; shift ;;
+      --warp-concurrency) WARP_CONCURRENCY="$1"; shift ;;
+      --ssh-user) SSH_USER="$1"; shift ;;
+      --ssh-port) SSH_PORT="$1"; shift ;;
+      --preflight) PREFLIGHT=1 ;;
+      --log-file) LOG_FILE="$1"; shift ;;
+      --dry-run) DRY_RUN=1 ;;
+      -y|--yes) ASSUME_YES=1 ;;
+      -h|--help) usage; exit 0 ;;
+      *) die "unknown option: ${opt} (see --help)" ;;
+    esac
+  done
+  if [ -n "${LOG_FILE}" ]; then
+    mkdir -p "$(dirname "${LOG_FILE}")"
+    exec > >(tee -a "${LOG_FILE}") 2>&1
+  fi
+  if [ "${all}" -eq 1 ]; then
+    SELECTED_STEPS=(1 2 3 4 5 6 7)
+  fi
+  if [ "${PREFLIGHT}" -eq 1 ]; then
+    preflight
+    if [ "${#SELECTED_STEPS[@]}" -eq 0 ]; then
+      log "preflight only; done"
+      exit 0
+    fi
+  fi
+  [ "${#SELECTED_STEPS[@]}" -gt 0 ] || die "no steps selected (--all / --step / --steps)"
+  log "nodes: ${NODES[*]}  ssh user: ${SSH_USER}  version: ${RUSTFS_VERSION}"
+  log "package: $(resolve_package_url)"
+  log "result dir: ${RESULT_DIR}"
+  [ "${DRY_RUN}" -eq 1 ] && warn "DRY-RUN mode: only printing the commands that would run"
+  run_steps
+  log "all done"
+}
+
+# Allow sourcing the file for unit tests without running main.
+if [ "${RUSTFS_PERF_SCRIPT_SOURCE_ONLY:-0}" != "1" ]; then
+  main "$@"
+fi

scripts/test/rustfs_performance_testing.sh

@@ -0,0 +1,239 @@
+#!/usr/bin/env bash
+#
+# rustfs-performance-testing.sh
+# RustFS 对象存储压测脚本（固定版）：测试方法 + 执行 + 结果解析
+#
+# 测试方法
+#   1) 方法：GET / PUT / MIXED（warp 默认混合负载 45% GET + 55% PUT）
+#   2) 对象尺寸：1KiB 4KiB 16KiB 128KiB 1MiB 4MiB 8MiB 16MiB 32MiB 64MiB
+#   3) 并发：64；单轮时长：5m；轮间 sleep：60s；GET 对象数：2500
+#   4) 顺序：GET 全部尺寸 -> PUT 全部尺寸 -> MIXED 全部尺寸
+#   5) 结果解析：每轮结束后自动解析 warp 输出，写入
+#      summary.tsv（机器可读）与 summary.md（Markdown 汇总表）
+#
+# 依赖：warp >= v1.6（MinIO warp），bash，awk/sed/grep
+# 说明：warp v1.6.1 的 put 不支持 --objects，脚本已自动处理（仅 get/mixed 传该参数）
+#
+# 环境变量覆盖（不传时使用固定默认值）：
+#   WARP_HOST WARP_ACCESS_KEY WARP_SECRET_KEY WARP_BUCKET
+#   WARP_CONCURRENCY WARP_DURATION WARP_GET_OBJECTS WARP_SLEEP_BETWEEN_ROUNDS
+#   WARP_RESULT_DIR
+#   WARP_METHODS WARP_SIZES   # 手动指定方法/尺寸（逗号或空格分隔），不传则全量
+
+set -u -o pipefail
+
+HOST="${WARP_HOST:-rustfs-node1:9000,rustfs-node2:9000,rustfs-node3:9000,rustfs-node4:9000}"
+ACCESS_KEY="${WARP_ACCESS_KEY:-rustfs@test}"
+SECRET_KEY="${WARP_SECRET_KEY:-rustfs@test}"
+BUCKET="${WARP_BUCKET:-warp-benchmark-bucket}"
+CONCURRENCY="${WARP_CONCURRENCY:-64}"
+DURATION="${WARP_DURATION:-5m}"
+GET_OBJECTS="${WARP_GET_OBJECTS:-2500}"
+SLEEP_BETWEEN_ROUNDS="${WARP_SLEEP_BETWEEN_ROUNDS:-60}"
+RESULT_DIR="${WARP_RESULT_DIR:-$(pwd)/warp-bench-results-$(date +%Y%m%d-%H%M%S)}"
+
+if [ -n "${WARP_SIZES:-}" ] && [ "${WARP_SIZES}" != "all" ] && [ "${WARP_SIZES}" != "ALL" ]; then
+  read -r -a SIZES <<<"${WARP_SIZES//,/ }"
+else
+  SIZES=(1KiB 4KiB 16KiB 128KiB 1MiB 4MiB 8MiB 16MiB 32MiB 64MiB)
+fi
+if [ -n "${WARP_METHODS:-}" ] && [ "${WARP_METHODS}" != "all" ] && [ "${WARP_METHODS}" != "ALL" ]; then
+  read -r -a METHODS <<<"${WARP_METHODS//,/ }"
+else
+  METHODS=(get put mixed)
+fi
+TOTAL_ROUNDS=$(( ${#METHODS[@]} * ${#SIZES[@]} ))
+ROUND=0
+
+# --parse-only <result-dir>：只解析已有结果目录（${method}_${size}.txt），不执行压测
+if [[ "${1:-}" == "--parse-only" && -n "${2:-}" ]]; then
+  RESULT_DIR="$2"
+fi
+
+LOG_FILE="${RESULT_DIR}/master.log"
+SUMMARY_TSV="${RESULT_DIR}/summary.tsv"
+SUMMARY_MD="${RESULT_DIR}/summary.md"
+
+if [[ "${1:-}" != "--parse-only" ]] && ! command -v warp >/dev/null 2>&1; then
+  echo "错误：未找到 warp 命令，请先安装 MinIO warp。" >&2
+  exit 1
+fi
+
+log() {
+  echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') $*" | tee -a "${LOG_FILE}"
+}
+
+# ---- 结果解析 ----
+
+# 提取指定 section（GET/PUT/Total 等）的 Average / Reqs / TTFB 原始行
+section_lines() {
+  awk -v sec="$2" '
+    /^Report: / { cur = $2; sub(/\.$/, "", cur) }
+    cur == sec && /^ *\* Average:/ { avg = $0 }
+    cur == sec && /^ *\* Reqs:/   { reqs = $0 }
+    cur == sec && /^ *\* TTFB:/   { ttfb = $0 }
+    END {
+      if (avg != "") print avg
+      if (reqs != "") print reqs
+      if (ttfb != "") print ttfb
+    }
+  ' "$1"
+}
+
+# 从统计行中取字段：tp objs avg p50 p90 p99 ttfb_avg ttfb_p99 ttfb_worst
+field() {
+  case "$2" in
+    tp)         echo "$1" | sed -n 's/^ *\* Average: \(.*\), \([0-9.]*\) obj\/s.*/\1/p' ;;
+    objs)       echo "$1" | sed -n 's/^ *\* Average: .*, \([0-9.]*\) obj\/s.*/\1/p' ;;
+    avg)        echo "$1" | sed -n 's/^ *\* Reqs: Avg: \([^,]*\),.*/\1/p' ;;
+    p50)        echo "$1" | sed -n 's/^ *\* Reqs: Avg: [^,]*, 50%: \([^,]*\),.*/\1/p' ;;
+    p90)        echo "$1" | sed -n 's/^ *\* Reqs: Avg: [^,]*, 50%: [^,]*, 90%: \([^,]*\),.*/\1/p' ;;
+    p99)        echo "$1" | sed -n 's/^ *\* Reqs: Avg: [^,]*, 50%: [^,]*, 90%: [^,]*, 99%: \([^,]*\),.*/\1/p' ;;
+    ttfb_avg)   echo "$1" | sed -n 's/^ *\* TTFB: Avg: \([^,]*\),.*/\1/p' ;;
+    ttfb_p99)   echo "$1" | sed -n 's/^ *\* TTFB: .*99th: \([^,]*\),.*/\1/p' ;;
+    ttfb_worst) echo "$1" | sed -n 's/^ *\* TTFB: .*Worst: \([^ ]*\).*/\1/p' ;;
+    *) echo "" ;;
+  esac
+}
+
+# 解析一轮输出，追加一行到 summary.tsv
+parse_round() {
+  local method="$1" size="$2" file="$3"
+  local line
+  if [[ "${method}" == "mixed" ]]; then
+    local total get put
+    total=$(section_lines "$file" Total)
+    get=$(section_lines "$file" GET)
+    put=$(section_lines "$file" PUT)
+    line=$(printf 'mixed\t%s\t%s\t%s\t%s\t%s' \
+      "$size" \
+      "$(field "${total}" tp)" \
+      "$(field "${total}" objs)" \
+      "$(field "${get}" avg)" \
+      "$(field "${put}" avg)")
+  else
+    local sec
+    sec=$(printf '%s' "${method}" | tr '[:lower:]' '[:upper:]')
+    local stats
+    stats=$(section_lines "$file" "${sec}")
+    line=$(printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s' \
+      "${method}" "${size}" \
+      "$(field "${stats}" tp)" \
+      "$(field "${stats}" objs)" \
+      "$(field "${stats}" avg)" \
+      "$(field "${stats}" p50)" \
+      "$(field "${stats}" p90)" \
+      "$(field "${stats}" p99)" \
+      "$(field "${stats}" ttfb_avg)" \
+      "$(field "${stats}" ttfb_p99)" \
+      "$(field "${stats}" ttfb_worst)")
+  fi
+  printf '%s\n' "${line}" >> "${SUMMARY_TSV}"
+}
+
+# 汇总 summary.tsv -> summary.md（Markdown 表格）
+gen_summary_md() {
+  {
+    echo "# RustFS 性能压测结果"
+    echo ""
+    echo "- 日期：$(date -u '+%Y-%m-%d %H:%M:%S UTC')"
+    echo "- 目标：${HOST}"
+    echo "- 并发：${CONCURRENCY}；单轮：${DURATION}；sleep：${SLEEP_BETWEEN_ROUNDS}s；GET objects：${GET_OBJECTS}"
+    echo "- 方法：GET / PUT / MIXED（warp 默认混合负载）；尺寸：${SIZES[*]}"
+    echo ""
+  } > "${SUMMARY_MD}"
+
+  for m in get put; do
+    {
+      echo "## $(printf '%s' "$m" | tr '[:lower:]' '[:upper:]') 结果"
+      echo ""
+      echo "| 对象尺寸 | 平均吞吐 | 平均 obj/s | Avg Latency | P50 | P90 | P99 | TTFB Avg | TTFB P99 | TTFB 最差 |"
+      echo "|----------|----------|-----------|-------------|-----|-----|-----|----------|----------|-----------|"
+    } >> "${SUMMARY_MD}"
+    while IFS=$'\t' read -r method size tp objs avg p50 p90 p99 ttfb_avg ttfb_p99 ttfb_worst; do
+      [[ "${method}" == "${m}" ]] && \
+        echo "| ${size} | ${tp} | ${objs} | ${avg} | ${p50} | ${p90} | ${p99} | ${ttfb_avg} | ${ttfb_p99} | ${ttfb_worst} |" >> "${SUMMARY_MD}"
+    done < "${SUMMARY_TSV}"
+    echo "" >> "${SUMMARY_MD}"
+  done
+
+  {
+    echo "## MIXED 结果（Total 口径）"
+    echo ""
+    echo "| 对象尺寸 | Total 平均吞吐 | Total 平均 obj/s | Mixed-GET Avg | Mixed-PUT Avg |"
+    echo "|----------|---------------|------------------|----------------|----------------|"
+  } >> "${SUMMARY_MD}"
+  while IFS=$'\t' read -r method size tp objs gavg pavg rest; do
+    [[ "${method}" == "mixed" ]] && \
+      echo "| ${size} | ${tp} | ${objs} | ${gavg} | ${pavg} |" >> "${SUMMARY_MD}"
+  done < "${SUMMARY_TSV}"
+  echo "" >> "${SUMMARY_MD}"
+}
+
+# ---- 主流程 ----
+
+mkdir -p "${RESULT_DIR}"
+log "CONFIG host=${HOST} bucket=${BUCKET} concurrency=${CONCURRENCY} duration=${DURATION} get_objects=${GET_OBJECTS} sleep_between_rounds=${SLEEP_BETWEEN_ROUNDS}s"
+printf 'method\tsize\tthroughput\tobj_per_s\treq_avg\treq_p50\treq_p90\treq_p99\tttfb_avg\tttfb_p99\tttfb_worst\n' > "${SUMMARY_TSV}"
+
+if [[ "${1:-}" == "--parse-only" ]]; then
+  for method in "${METHODS[@]}"; do
+    for size in "${SIZES[@]}"; do
+      outfile="${RESULT_DIR}/${method}_${size}.txt"
+      if [[ -s "${outfile}" ]]; then
+        parse_round "${method}" "${size}" "${outfile}"
+      fi
+    done
+  done
+  gen_summary_md
+  echo "parsed from ${RESULT_DIR}"
+  echo ""
+  cat "${SUMMARY_MD}"
+  exit 0
+fi
+
+for method in "${METHODS[@]}"; do
+  for size in "${SIZES[@]}"; do
+    ROUND=$((ROUND + 1))
+    outfile="${RESULT_DIR}/${method}_${size}.txt"
+
+    log "START round=${ROUND}/${TOTAL_ROUNDS} method=${method} size=${size} concurrency=${CONCURRENCY} duration=${DURATION}"
+
+    extra_args=()
+    if [[ "${method}" != "put" ]]; then
+      extra_args=(--objects "${GET_OBJECTS}")
+    fi
+
+    start_epoch=$(date +%s)
+    warp "${method}" \
+      --host "${HOST}" \
+      --access-key "${ACCESS_KEY}" \
+      --secret-key "${SECRET_KEY}" \
+      --bucket "${BUCKET}" \
+      --concurrent "${CONCURRENCY}" \
+      --duration "${DURATION}" \
+      --obj.size "${size}" \
+      "${extra_args[@]}" \
+      --no-color 2>&1 | tee "${outfile}"
+    rc=${PIPESTATUS[0]}
+    end_epoch=$(date +%s)
+
+    if [[ ${rc} -eq 0 ]]; then
+      parse_round "${method}" "${size}" "${outfile}"
+      log "END round=${ROUND}/${TOTAL_ROUNDS} method=${method} size=${size} rc=${rc} elapsed=$((end_epoch - start_epoch))s parsed=ok"
+    else
+      log "END round=${ROUND}/${TOTAL_ROUNDS} method=${method} size=${size} rc=${rc} elapsed=$((end_epoch - start_epoch))s parsed=skipped"
+    fi
+
+    if [[ ${ROUND} -lt ${TOTAL_ROUNDS} ]]; then
+      log "SLEEP ${SLEEP_BETWEEN_ROUNDS}s before next round"
+      sleep "${SLEEP_BETWEEN_ROUNDS}"
+    fi
+  done
+done
+
+gen_summary_md
+log "ALL_ROUNDS_COMPLETE summary_tsv=${SUMMARY_TSV} summary_md=${SUMMARY_MD}"
+echo ""
+echo "==== 结果汇总 ===="
+cat "${SUMMARY_MD}"

scripts/test/rustfs_pool_expand.sh

@@ -93,6 +93,9 @@ WARP_BUCKET="test-10mb"
 WARP_OBJ_SIZE="100MiB"
 WARP_CONCURRENT=32
 WARP_DURATION="5m"
+# Warp log path; empty = auto-created unique temp file (the runner user may
+# not be able to write a shared /tmp path owned by another user).
+WARP_LOG_FILE="${RUSTFS_WARP_LOG_FILE:-}"
 STORAGE_THRESHOLD=85            # stop writing when usage reaches N% (note suggests 80-85)
 POLL_INTERVAL=30                # status polling interval (seconds)
 
@@ -102,6 +105,8 @@ DECOMMISSION_TIMEOUT=86400
 SERVICE_TIMEOUT=300
 DECOMMISSION_RETRIES=3          # auto clear+retry attempts after a failed decommission
 DECOMMISSION_RETRY_DELAY=30     # delay between retries (seconds)
+REBALANCE_START_RETRIES=6       # rebalance start retries (fleet proof may take ~10-20s after a topology change)
+REBALANCE_START_RETRY_DELAY=20  # delay between rebalance start retries (seconds)
 
 # Pool to decommission (zero-based; 0 in the note)
 DECOMMISSION_POOL_ID=0
@@ -325,9 +330,45 @@ wait_service_active() {
     sleep 5
     waited=$((waited + 5))
   done
+  diagnose_node_start_failure "${node}"
   die "${node}: timed out waiting for ${RUSTFS_SERVICE} (${SERVICE_TIMEOUT}s)"
 }
 
+# Known server-side issues the test can hit. Format:
+#   "<error signature>|<tracking>|<hint>"
+KNOWN_SERVER_ISSUES=(
+  "pool activation requires a live fleet capability proof|rustfs/backlog#2031|server-side cold-start recovery is covered by this PR; if this appears, collect node journals and treat it as a regression"
+)
+
+# Print a hint when $1 matches a known server-side issue signature.
+hint_server_issue() {
+  local text="$1" entry sig tracking hint
+  for entry in "${KNOWN_SERVER_ISSUES[@]}"; do
+    sig="${entry%%|*}"
+    tracking="${entry#*|}"
+    hint="${tracking#*|}"
+    tracking="${tracking%%|*}"
+    if printf '%s' "${text}" | grep -qiF "${sig}"; then
+      printf '\033[1;33m[KNOWN SERVER ISSUE]\033[0m %s (%s): %s\n' "${sig}" "${tracking}" "${hint}" >&2
+      return 0
+    fi
+  done
+  return 1
+}
+
+# Fetch the journal tail from a node whose service failed to start and
+# annotate known server-side issues.
+diagnose_node_start_failure() {
+  local node="$1" journal
+  if ! journal="$(ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" \
+    "SUDO=\"\"; [ \"\$(id -u)\" -ne 0 ] && SUDO=\"sudo -n\"; \${SUDO} journalctl -u ${RUSTFS_SERVICE} --no-pager -n 60 2>/dev/null || true")"; then
+    journal="unable to collect journal (SSH command failed)"
+  fi
+  printf '%s\n' "--- ${node}: ${RUSTFS_SERVICE} journal (last 60 lines) ---" >&2
+  printf '%s\n' "${journal}" >&2
+  hint_server_issue "${journal}" || true
+}
+
 # Generate the /etc/default/rustfs content
 rustfs_config_body() {
   local volumes="$1"
@@ -406,9 +447,13 @@ service_action() {
   local action="$1" node="$2"
   log "${node}: systemctl ${action} ${RUSTFS_SERVICE}"
   if [ "${DRY_RUN}" -eq 1 ]; then return 0; fi
-  ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" \
-    "if [ \"\$(id -u)\" -ne 0 ]; then sudo -n systemctl ${action} ${RUSTFS_SERVICE}; else systemctl ${action} ${RUSTFS_SERVICE}; fi" \
-    || die "${node}: systemctl ${action} failed"
+  if ! ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" \
+      "if [ \"\$(id -u)\" -ne 0 ]; then sudo -n systemctl ${action} ${RUSTFS_SERVICE}; else systemctl ${action} ${RUSTFS_SERVICE}; fi"; then
+    if [ "${action}" = "start" ]; then
+      diagnose_node_start_failure "${node}"
+    fi
+    die "${node}: systemctl ${action} failed"
+  fi
 }
 
 service_action_all() {
@@ -587,6 +632,29 @@ wait_rebalance() {
   die "timed out waiting for rebalance (${REBALANCE_TIMEOUT}s)"
 }
 
+# Start rebalance via the admin API. Nightly builds gate rebalance activation
+# on a live cross-pool fence fleet capability proof that is re-established
+# shortly after a pool joins, so retry a few times before failing.
+start_rebalance_with_retry() {
+  local attempts="${REBALANCE_START_RETRIES}" delay="${REBALANCE_START_RETRY_DELAY}" attempt=1 body code id
+  while :; do
+    body="$(admin_api POST /rustfs/admin/v3/rebalance/start "")"
+    code="$(admin_api_code)"
+    if [ "${code}" = "200" ]; then
+      id="$(printf '%s' "${body}" | jq -r '.id // empty')"
+      log "rebalance started: id=${id}"
+      return 0
+    fi
+    warn "rebalance start attempt ${attempt}/${attempts} failed (HTTP ${code}): ${body}"
+    if [ "${attempt}" -ge "${attempts}" ]; then
+      hint_server_issue "${body}" || true
+      die "rebalance start failed after ${attempts} attempts (see last error above)"
+    fi
+    attempt=$((attempt + 1))
+    sleep "${delay}"
+  done
+}
+
 # Print a detailed decommission failure/progress report for one pool
 print_decommission_detail() {
   local body="$1" pool_id="$2" label="$3"
@@ -817,16 +885,18 @@ step4_write_data() {
       log "DRY-RUN: monitoring storage usage until ${STORAGE_THRESHOLD}%"
       return 0
     fi
-    log "starting warp writes (background)..."
+    local warp_log warp_pid
+    warp_log="${WARP_LOG_FILE:-$(mktemp "${TMPDIR:-/tmp}/rustfs-warp.XXXXXX.log")}"
+    log "starting warp writes (background), log: ${warp_log}"
     warp put --host "${API_ENDPOINT#http://}" \
       --bucket "${WARP_BUCKET}" \
       --access-key "${ACCESS_KEY}" \
       --secret-key "${SECRET_KEY}" \
       --obj.size "${WARP_OBJ_SIZE}" \
       --concurrent "${WARP_CONCURRENT}" \
-      --noprefix --duration "${WARP_DURATION}" --noclear >/tmp/rustfs-warp.log 2>&1 &
-    local warp_pid=$!
-    log "warp PID=${warp_pid}, log /tmp/rustfs-warp.log"
+      --noprefix --duration "${WARP_DURATION}" --noclear >"${warp_log}" 2>&1 &
+    warp_pid=$!
+    log "warp PID=${warp_pid}"
     trap 'kill "${warp_pid:-}" 2>/dev/null || true' EXIT
     monitor_storage "${STORAGE_THRESHOLD}" "${warp_pid}"
     kill "${warp_pid}" 2>/dev/null || true
@@ -866,12 +936,8 @@ step5_expand_pool2() {
 step6_rebalance() {
   log "step 6: start data rebalance (admin API)"
   confirm "About to start rebalance (POST ${API_ENDPOINT}/rustfs/admin/v3/rebalance/start). Continue?"
-  local body id
   if [ "${DRY_RUN}" -eq 0 ]; then
-    body="$(admin_api POST /rustfs/admin/v3/rebalance/start "")"
-    [ "$(admin_api_code)" = "200" ] || die "rebalance start failed (HTTP $(admin_api_code)): ${body}"
-    id="$(printf '%s' "${body}" | jq -r '.id // empty')"
-    log "rebalance started: id=${id}"
+    start_rebalance_with_retry
   fi
   wait_rebalance
 }
@@ -894,12 +960,8 @@ step7_expand_pool3() {
 step8_rebalance() {
   log "step 8: start data rebalance (admin API)"
   confirm "About to start rebalance (POST ${API_ENDPOINT}/rustfs/admin/v3/rebalance/start). Continue?"
-  local body id
   if [ "${DRY_RUN}" -eq 0 ]; then
-    body="$(admin_api POST /rustfs/admin/v3/rebalance/start "")"
-    [ "$(admin_api_code)" = "200" ] || die "rebalance start failed (HTTP $(admin_api_code)): ${body}"
-    id="$(printf '%s' "${body}" | jq -r '.id // empty')"
-    log "rebalance started: id=${id}"
+    start_rebalance_with_retry
   fi
   wait_rebalance
 }
@@ -928,6 +990,7 @@ step9_decommission() {
       break
     fi
     if [ "${attempt}" -ge "${DECOMMISSION_RETRIES}" ]; then
+      warn "if the source bucket has many objects and the tested version is 1.0.0-rc.3, this is the known metacache-listing decommission bug; remove the test bucket (rc rb --force rustfs/${WARP_BUCKET}) or lower --storage-threshold, then re-run step 9"
       die "pool ${DECOMMISSION_POOL_ID} still failed after ${attempt} attempts; investigate manually (POST ${API_ENDPOINT}/rustfs/admin/v3/pools/clear?by-id=true&pool=${DECOMMISSION_POOL_ID} to reset)"
     fi
     warn "attempt ${attempt} failed; clearing metadata and retrying in ${DECOMMISSION_RETRY_DELAY}s"
@@ -1065,6 +1128,12 @@ main() {
   trap 'rm -f "${ADMIN_API_CODE_FILE}"' EXIT
   if [ -n "${LOG_FILE}" ]; then
     mkdir -p "$(dirname "${LOG_FILE}")"
+    if ! touch "${LOG_FILE}" 2>/dev/null; then
+      # A fixed /tmp path may be owned by another user (e.g. a previous root
+      # run); fall back to a unique, always-writable temp file.
+      LOG_FILE="$(mktemp "${TMPDIR:-/tmp}/rustfs-pool-test.XXXXXX.log")"
+      warn "log file not writable; using ${LOG_FILE}"
+    fi
     exec > >(tee -a "${LOG_FILE}") 2>&1
   fi
   if [ "${RESET}" -eq 1 ]; then