Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a814bf3ae3 | ||
|
|
d628a2f48b | ||
|
|
2bd1d70729 | ||
|
|
0b96a992d6 | ||
|
|
bbc96c43a2 | ||
|
|
ec9aabcf00 |
@@ -6,7 +6,7 @@ description: "Run the end-to-end RustFS console gate, version bump, preview vali
|
||||
|
||||
This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
|
||||
|
||||
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. That Release is temporary: `build.yml` deletes it automatically once the final tag's Release is published, so the Releases page ends up carrying deliverables only while the `-preview.N` tags stay behind as the traceability record. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
|
||||
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
|
||||
|
||||
Pipeline shape:
|
||||
|
||||
@@ -19,7 +19,6 @@ check console main against its latest Release
|
||||
-> validate with latest rc client
|
||||
-> report preview acceptance results -> STOP for explicit human confirmation
|
||||
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
|
||||
-> CI deletes the <target>-preview.N Releases (tags kept)
|
||||
```
|
||||
|
||||
On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
|
||||
@@ -52,16 +51,14 @@ Rules:
|
||||
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
|
||||
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
|
||||
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
|
||||
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
|
||||
|
||||
## Hard rules
|
||||
|
||||
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
|
||||
- Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
|
||||
- Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
|
||||
- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
|
||||
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
|
||||
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
|
||||
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
|
||||
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
|
||||
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
|
||||
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
|
||||
- Completing preview acceptance does not authorize the final tag. After Phases 3–5 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
|
||||
@@ -233,7 +230,6 @@ git push origin "<target>"
|
||||
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
|
||||
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
|
||||
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
|
||||
- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
|
||||
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
|
||||
|
||||
## Output contract
|
||||
@@ -243,5 +239,5 @@ Always report:
|
||||
- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
|
||||
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
|
||||
- Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
|
||||
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
|
||||
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
|
||||
- Any deviation from this pipeline and why the user approved it.
|
||||
|
||||
@@ -5,4 +5,3 @@ self-hosted-runner:
|
||||
- sm-standard-2
|
||||
- sm-standard-4
|
||||
- dind-sm-standard-2
|
||||
- smoke-testing
|
||||
|
||||
@@ -1033,55 +1033,6 @@ jobs:
|
||||
echo "🎉 Released $TAG successfully!"
|
||||
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
|
||||
|
||||
# Remove the internal preview releases once the deliverable release is live.
|
||||
# Only the Releases are deleted; the -preview.N tags stay so the validated
|
||||
# commit remains traceable.
|
||||
cleanup-preview-releases:
|
||||
name: Cleanup Preview Releases
|
||||
needs: [ build-check, publish-release ]
|
||||
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: write
|
||||
steps:
|
||||
- name: Delete preview releases for this target
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
TAG="${{ needs.build-check.outputs.version }}"
|
||||
RELEASES_JSON="${RUNNER_TEMP}/releases.json"
|
||||
|
||||
# Fetch before filtering: a failed listing must abort here instead of
|
||||
# looking like "nothing to clean up".
|
||||
gh api --paginate "repos/${GITHUB_REPOSITORY}/releases?per_page=100" > "$RELEASES_JSON"
|
||||
|
||||
# Match only <target>-preview.<digits>. String operations, not a
|
||||
# regex over the tag, so dots in the version cannot widen the match.
|
||||
DELETED=0
|
||||
while IFS= read -r preview_tag; do
|
||||
[[ -n "$preview_tag" ]] || continue
|
||||
echo "🧹 Deleting preview release $preview_tag (tag kept)"
|
||||
gh release delete "$preview_tag" --yes
|
||||
DELETED=$((DELETED + 1))
|
||||
done < <(
|
||||
jq -r --arg tag "$TAG" '
|
||||
.[]
|
||||
| select(.tag_name | startswith($tag + "-preview."))
|
||||
| select(.tag_name | ltrimstr($tag + "-preview.") | test("^[0-9]+$"))
|
||||
| .tag_name
|
||||
' "$RELEASES_JSON"
|
||||
)
|
||||
|
||||
if [[ "$DELETED" -eq 0 ]]; then
|
||||
echo "ℹ️ No preview releases to clean up for $TAG"
|
||||
else
|
||||
echo "✅ Removed $DELETED preview release(s) for $TAG"
|
||||
fi
|
||||
|
||||
alert-on-failure:
|
||||
name: Alert on scheduled failure
|
||||
needs: [build-check, prepare-platform-matrix, build-rustfs, build-summary]
|
||||
|
||||
@@ -1,126 +0,0 @@
|
||||
name: RustFS Heal Test
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
package_url:
|
||||
description: 'Direct .deb URL (nightly/R2). Defaults to the latest nightly deb.'
|
||||
required: false
|
||||
type: string
|
||||
stop_node_gb:
|
||||
description: 'Stop the outage node when surviving nodes reach N GiB'
|
||||
required: false
|
||||
default: '15'
|
||||
warp_stop_gb:
|
||||
description: 'Stop warp when surviving nodes reach N GiB'
|
||||
required: false
|
||||
default: '40'
|
||||
heal_target_gb:
|
||||
description: 'Outage node must reach N GiB after heal to pass'
|
||||
required: false
|
||||
default: '40'
|
||||
cleanup_before:
|
||||
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
|
||||
type: boolean
|
||||
default: true
|
||||
cleanup_after:
|
||||
description: 'Reset the nodes after the test (DESTROYS test data/config)'
|
||||
type: boolean
|
||||
default: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# Only one test at a time: both this and the pool-expansion workflow mutate
|
||||
# the same test environment, so they share one concurrency group.
|
||||
concurrency:
|
||||
group: rustfs-pool-expansion-test
|
||||
cancel-in-progress: false
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
|
||||
env:
|
||||
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
|
||||
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
|
||||
RUSTFS_API_ENDPOINT: ${{ secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }}
|
||||
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
|
||||
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
|
||||
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
|
||||
|
||||
jobs:
|
||||
heal-test:
|
||||
runs-on: smoke-testing
|
||||
timeout-minutes: 480
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Show environment
|
||||
run: |
|
||||
uname -a
|
||||
jq --version
|
||||
openssl version
|
||||
warp --version || true
|
||||
df -h /data | tail -1
|
||||
|
||||
- name: Reset test environment (before)
|
||||
if: ${{ inputs.cleanup_before != 'false' }}
|
||||
run: |
|
||||
chmod +x scripts/test/rustfs_heal_test.sh
|
||||
./scripts/test/rustfs_heal_test.sh --reset -y
|
||||
|
||||
- name: Install RustFS package & start cluster
|
||||
run: |
|
||||
ARGS=(--steps "1,2" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./scripts/test/rustfs_heal_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Preflight checks
|
||||
run: |
|
||||
ARGS=(--preflight --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./scripts/test/rustfs_heal_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Run heal test (write -> outage -> heal -> verify)
|
||||
run: |
|
||||
./scripts/test/rustfs_heal_test.sh \
|
||||
--steps "3,4,5,6,7" -y \
|
||||
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
|
||||
--stop-node-gb "${{ inputs.stop_node_gb }}" \
|
||||
--warp-stop-gb "${{ inputs.warp_stop_gb }}" \
|
||||
--heal-target-gb "${{ inputs.heal_target_gb }}" \
|
||||
--log-file /tmp/rustfs-heal-test.log
|
||||
|
||||
- name: Upload test logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
with:
|
||||
name: rustfs-heal-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-heal-test*.log
|
||||
/tmp/rustfs-warp.*.log
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Reset test environment (after)
|
||||
if: ${{ always() && inputs.cleanup_after != 'false' }}
|
||||
run: |
|
||||
./scripts/test/rustfs_heal_test.sh --reset -y
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: |
|
||||
echo "RustFS heal test failed"
|
||||
echo "Package source: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
|
||||
echo "See the uploaded log artifact for details."
|
||||
@@ -30,18 +30,6 @@ on:
|
||||
description: 'Run the pool decommission step (3-pool topology only)'
|
||||
type: boolean
|
||||
default: true
|
||||
stop_node_gb:
|
||||
description: 'Heal: stop the outage node when surviving nodes reach N GiB'
|
||||
required: false
|
||||
default: '15'
|
||||
warp_stop_gb:
|
||||
description: 'Heal: stop warp when surviving nodes reach N GiB'
|
||||
required: false
|
||||
default: '40'
|
||||
heal_target_gb:
|
||||
description: 'Heal: outage node must reach N GiB after heal'
|
||||
required: false
|
||||
default: '40'
|
||||
cleanup_before:
|
||||
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
|
||||
type: boolean
|
||||
@@ -50,10 +38,9 @@ on:
|
||||
description: 'Reset the nodes after the test (DESTROYS test data/config)'
|
||||
type: boolean
|
||||
default: true
|
||||
workflow_run:
|
||||
# Run after the nightly build completes: pool expansion first, then heal.
|
||||
workflows: ["Nightly GNU Build"]
|
||||
types: [completed]
|
||||
schedule:
|
||||
# Nightly regression run; remove if you do not want a schedule.
|
||||
- cron: '0 21 * * *'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -74,23 +61,19 @@ env:
|
||||
RUSTFS_API_ENDPOINT: ${{ secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }}
|
||||
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
|
||||
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
|
||||
# Package used by the nightly run (workflow_dispatch inputs are empty for
|
||||
# workflow_run events), i.e. the latest nightly deb published by nightly-gnu.yml.
|
||||
# Package used by the scheduled run (workflow_dispatch inputs are empty for
|
||||
# schedule events), i.e. the latest nightly deb published by nightly-gnu.yml.
|
||||
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
|
||||
|
||||
jobs:
|
||||
pool-expansion-test:
|
||||
runs-on: smoke-testing
|
||||
timeout-minutes: 360
|
||||
# Run on manual dispatch, or when the nightly build completed successfully
|
||||
# (its deb is what the tests install). Skipped when nightly failed.
|
||||
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
ref: ${{ github.event.workflow_run.head_sha || github.ref }}
|
||||
|
||||
- name: Show environment
|
||||
run: |
|
||||
@@ -108,7 +91,7 @@ jobs:
|
||||
|
||||
- name: Install RustFS package & start first pool
|
||||
run: |
|
||||
ARGS=(--steps "1,2,3" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
ARGS=(--steps 1,2,3 -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
elif [ -n "${{ inputs.rustfs_version }}" ]; then
|
||||
@@ -154,8 +137,8 @@ jobs:
|
||||
with:
|
||||
name: rustfs-pool-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-pool-test*.log
|
||||
/tmp/rustfs-warp.*.log
|
||||
/tmp/rustfs-pool-test.log
|
||||
/tmp/rustfs-warp.log
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Reset test environment (after)
|
||||
@@ -169,75 +152,3 @@ jobs:
|
||||
echo "RustFS pool expansion test failed"
|
||||
echo "Package source: ${{ inputs.package_url || inputs.rustfs_version || 'nightly (R2 latest)' }}"
|
||||
echo "See the uploaded log artifact for details."
|
||||
|
||||
# Heal regression runs after the pool test regardless of its outcome: a pool
|
||||
# failure must be reported (it makes the run red) but must not block heal.
|
||||
heal-test:
|
||||
name: Heal test (after pool test)
|
||||
runs-on: smoke-testing
|
||||
timeout-minutes: 480
|
||||
needs: pool-expansion-test
|
||||
if: ${{ always() && (github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success') }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
ref: ${{ github.event.workflow_run.head_sha || github.ref }}
|
||||
|
||||
- name: Reset test environment (before)
|
||||
if: ${{ inputs.cleanup_before != 'false' }}
|
||||
run: |
|
||||
chmod +x scripts/test/rustfs_heal_test.sh
|
||||
./scripts/test/rustfs_heal_test.sh --reset -y
|
||||
|
||||
- name: Install RustFS package & start cluster
|
||||
run: |
|
||||
ARGS=(--steps "1,2" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./scripts/test/rustfs_heal_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Preflight checks
|
||||
run: |
|
||||
ARGS=(--preflight --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./scripts/test/rustfs_heal_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Run heal test (write -> outage -> heal -> verify)
|
||||
run: |
|
||||
./scripts/test/rustfs_heal_test.sh \
|
||||
--steps 3,4,5,6,7 -y \
|
||||
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
|
||||
--stop-node-gb "${{ inputs.stop_node_gb || '15' }}" \
|
||||
--warp-stop-gb "${{ inputs.warp_stop_gb || '40' }}" \
|
||||
--heal-target-gb "${{ inputs.heal_target_gb || '40' }}" \
|
||||
--log-file /tmp/rustfs-heal-test.log
|
||||
|
||||
- name: Upload test logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
with:
|
||||
name: rustfs-heal-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-heal-test.log
|
||||
/tmp/rustfs-warp.*.log
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Reset test environment (after)
|
||||
if: ${{ always() && inputs.cleanup_after != 'false' }}
|
||||
run: |
|
||||
./scripts/test/rustfs_heal_test.sh --reset -y
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: |
|
||||
echo "RustFS heal test failed"
|
||||
echo "See the uploaded log artifact for details."
|
||||
|
||||
Generated
+4
@@ -91,6 +91,7 @@ dependencies = [
|
||||
"const-random",
|
||||
"getrandom 0.3.4",
|
||||
"once_cell",
|
||||
"serde",
|
||||
"version_check",
|
||||
"zerocopy",
|
||||
]
|
||||
@@ -9626,6 +9627,7 @@ dependencies = [
|
||||
name = "rustfs-ecstore"
|
||||
version = "1.0.0-rc.4"
|
||||
dependencies = [
|
||||
"ahash",
|
||||
"arc-swap",
|
||||
"async-channel",
|
||||
"async-recursion",
|
||||
@@ -9771,6 +9773,7 @@ dependencies = [
|
||||
name = "rustfs-filemeta"
|
||||
version = "1.0.0-rc.4"
|
||||
dependencies = [
|
||||
"ahash",
|
||||
"arc-swap",
|
||||
"byteorder",
|
||||
"bytes",
|
||||
@@ -10811,6 +10814,7 @@ dependencies = [
|
||||
name = "rustfs-utils"
|
||||
version = "1.0.0-rc.4"
|
||||
dependencies = [
|
||||
"ahash",
|
||||
"base64-simd",
|
||||
"blake2",
|
||||
"brotli",
|
||||
|
||||
@@ -368,6 +368,9 @@ hotpath = { version = "0.24.0", default-features = false }
|
||||
# Snapshot testing for output format regression detection
|
||||
insta = { version = "1.48" }
|
||||
|
||||
# High-performance hashing
|
||||
ahash = { version = "0.8", default-features = false, features = ["std", "runtime-rng", "serde"] }
|
||||
|
||||
[workspace.metadata.cargo-shear]
|
||||
ignored = ["hotpath", "rustfs"]
|
||||
|
||||
|
||||
@@ -40,14 +40,6 @@ pub const HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS: u64 = 5_000;
|
||||
pub const ENV_HEALTH_CLUSTER_TIMEOUT_MS: &str = "RUSTFS_HEALTH_CLUSTER_TIMEOUT_MS";
|
||||
pub const DEFAULT_HEALTH_CLUSTER_TIMEOUT_MS: u64 = 2000;
|
||||
|
||||
/// Timeout for one remote lock-client online check used by readiness (milliseconds).
|
||||
///
|
||||
/// This is intentionally shorter than the generic lock RPC timeout so
|
||||
/// `/health/ready` can report degradation instead of riding a dead peer's
|
||||
/// connect or HTTP/2 keepalive budget.
|
||||
pub const ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS: &str = "RUSTFS_HEALTH_LOCK_ONLINE_TIMEOUT_MS";
|
||||
pub const DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS: u64 = 1000;
|
||||
|
||||
/// Maximum time to wait for local node runtime readiness (storage / IAM / lock
|
||||
/// quorum) during startup before failing fast (seconds).
|
||||
///
|
||||
|
||||
@@ -288,39 +288,6 @@ pub const DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 0;
|
||||
|
||||
const _: () = assert!(!DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE);
|
||||
|
||||
/// Enable automatic foreground admission for large or unknown-size PutObject requests.
|
||||
///
|
||||
/// Unlike the strict experimental gate above, this default-on path only applies
|
||||
/// to requests that are large enough to create sustained erasure/RPC pressure.
|
||||
/// Small PUTs continue on the legacy path unless the strict gate is explicitly
|
||||
/// enabled.
|
||||
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE";
|
||||
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: bool = true;
|
||||
|
||||
/// Maximum large foreground PutObject requests admitted concurrently per process.
|
||||
///
|
||||
/// `0` derives a conservative default from the local disk-read scheduler cap,
|
||||
/// currently clamped to protect the commit path without making ordinary high
|
||||
/// throughput uploads single-file.
|
||||
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT";
|
||||
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: usize = 0;
|
||||
|
||||
/// Minimum object size that enters automatic large PutObject admission.
|
||||
///
|
||||
/// Requests with an unknown size are treated as large because the write pressure
|
||||
/// cannot be bounded from headers.
|
||||
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES";
|
||||
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 32 * 1024 * 1024;
|
||||
|
||||
/// Time in milliseconds a large foreground PutObject waits for a permit.
|
||||
///
|
||||
/// A short wait smooths transient bursts while still returning S3
|
||||
/// `SlowDown`/503 before body ingest when the node is already saturated.
|
||||
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
|
||||
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 250;
|
||||
|
||||
const _: () = assert!(DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE);
|
||||
|
||||
/// Environment variable for minimum GetObject timeout in seconds.
|
||||
///
|
||||
/// When dynamic timeout calculation is enabled, this is the minimum timeout
|
||||
|
||||
@@ -1,220 +0,0 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Anonymous access to SSE-KMS objects under per-key authorization.
|
||||
//!
|
||||
//! Locks both halves of the anonymous contract decided in backlog#2028 (D4):
|
||||
//!
|
||||
//! - **Enforcement on**: anonymous requests hold no `kms` grants, so a public
|
||||
//! bucket policy does not let them read SSE-KMS objects or write through an
|
||||
//! SSE-KMS default-encryption rule. Both fail with `AccessDenied`.
|
||||
//! - **Enforcement off** (the default): bucket policy alone governs anonymous
|
||||
//! access, matching the pre-enforcement behavior — public SSE-KMS objects are
|
||||
//! decrypted and served, and anonymous writes are encrypted under the default
|
||||
//! key.
|
||||
//!
|
||||
//! The denial today is emergent — an empty-account principal falling through to
|
||||
//! the IAM default deny — so without this file a refactor of principal
|
||||
//! construction or policy evaluation could silently flip it. Each test carries a
|
||||
//! plaintext-object positive control: a denial proves nothing while the bucket
|
||||
//! policy has not propagated.
|
||||
|
||||
use super::common::{LocalKMSTestEnvironment, create_key_with_specific_id};
|
||||
use crate::common::{init_logging, local_http_client};
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use aws_sdk_s3::types::{
|
||||
ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
|
||||
};
|
||||
use std::time::Duration;
|
||||
|
||||
type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
|
||||
|
||||
const DEFAULT_KEY: &str = "kms-anon-default-key";
|
||||
const BUCKET: &str = "kms-anon-enforcement";
|
||||
const PLAIN_OBJECT: &str = "plain.txt";
|
||||
const ENCRYPTED_OBJECT: &str = "encrypted.txt";
|
||||
const PAYLOAD: &[u8] = b"kms anonymous enforcement payload";
|
||||
|
||||
/// How long a bucket policy change may take to reach the request path.
|
||||
const POLICY_PROPAGATION: Duration = Duration::from_secs(20);
|
||||
|
||||
/// Start a local-KMS server and build the public-bucket fixture.
|
||||
///
|
||||
/// The bucket holds a plaintext object (the positive control), an SSE-KMS
|
||||
/// object, an SSE-KMS default-encryption rule, and a bucket policy opening
|
||||
/// `GetObject`/`PutObject` to everyone. The enforcement switch defaults to off,
|
||||
/// so the enforcing case has to set it explicitly.
|
||||
async fn start_public_sse_kms_bucket(env: &mut LocalKMSTestEnvironment, enforce: bool) -> TestResult {
|
||||
create_key_with_specific_id(&env.kms_keys_dir, DEFAULT_KEY).await?;
|
||||
|
||||
let key_dir = env.kms_keys_dir.clone();
|
||||
let args = vec![
|
||||
"--kms-enable",
|
||||
"--kms-backend",
|
||||
"local",
|
||||
"--kms-key-dir",
|
||||
key_dir.as_str(),
|
||||
"--kms-default-key-id",
|
||||
DEFAULT_KEY,
|
||||
];
|
||||
let mut envs = vec![("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")];
|
||||
if enforce {
|
||||
envs.push(("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "true"));
|
||||
}
|
||||
env.base_env.start_rustfs_server_with_env(args, &envs).await?;
|
||||
env.base_env.create_test_bucket(BUCKET).await?;
|
||||
|
||||
let owner = env.base_env.create_s3_client();
|
||||
|
||||
owner
|
||||
.put_object()
|
||||
.bucket(BUCKET)
|
||||
.key(PLAIN_OBJECT)
|
||||
.body(ByteStream::from_static(PAYLOAD))
|
||||
.send()
|
||||
.await?;
|
||||
owner
|
||||
.put_object()
|
||||
.bucket(BUCKET)
|
||||
.key(ENCRYPTED_OBJECT)
|
||||
.body(ByteStream::from_static(PAYLOAD))
|
||||
.server_side_encryption(ServerSideEncryption::AwsKms)
|
||||
.ssekms_key_id(DEFAULT_KEY)
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
let encryption_config = ServerSideEncryptionConfiguration::builder()
|
||||
.rules(
|
||||
ServerSideEncryptionRule::builder()
|
||||
.apply_server_side_encryption_by_default(
|
||||
ServerSideEncryptionByDefault::builder()
|
||||
.sse_algorithm(ServerSideEncryption::AwsKms)
|
||||
.kms_master_key_id(DEFAULT_KEY)
|
||||
.build()?,
|
||||
)
|
||||
.build(),
|
||||
)
|
||||
.build()?;
|
||||
owner
|
||||
.put_bucket_encryption()
|
||||
.bucket(BUCKET)
|
||||
.server_side_encryption_configuration(encryption_config)
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
let policy = serde_json::json!({
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [{
|
||||
"Sid": "PublicReadWrite",
|
||||
"Effect": "Allow",
|
||||
"Principal": "*",
|
||||
"Action": ["s3:GetObject", "s3:PutObject"],
|
||||
"Resource": [format!("arn:aws:s3:::{BUCKET}/*")]
|
||||
}]
|
||||
})
|
||||
.to_string();
|
||||
owner.put_bucket_policy().bucket(BUCKET).policy(&policy).send().await?;
|
||||
let _ = owner.delete_public_access_block().bucket(BUCKET).send().await;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn object_url(env: &LocalKMSTestEnvironment, key: &str) -> String {
|
||||
format!("{}/{BUCKET}/{key}", env.base_env.url)
|
||||
}
|
||||
|
||||
async fn anonymous_get(env: &LocalKMSTestEnvironment, key: &str) -> Result<reqwest::Response, reqwest::Error> {
|
||||
local_http_client().get(object_url(env, key)).send().await
|
||||
}
|
||||
|
||||
async fn anonymous_put(env: &LocalKMSTestEnvironment, key: &str) -> Result<reqwest::Response, reqwest::Error> {
|
||||
local_http_client().put(object_url(env, key)).body(PAYLOAD).send().await
|
||||
}
|
||||
|
||||
/// Retry the plaintext read until the public bucket policy is live.
|
||||
async fn wait_for_public_read(env: &LocalKMSTestEnvironment) -> TestResult {
|
||||
let deadline = tokio::time::Instant::now() + POLICY_PROPAGATION;
|
||||
loop {
|
||||
let status = anonymous_get(env, PLAIN_OBJECT).await?.status();
|
||||
if status.as_u16() == 200 {
|
||||
return Ok(());
|
||||
}
|
||||
if tokio::time::Instant::now() >= deadline {
|
||||
return Err(format!("positive control never became readable: anonymous GET {PLAIN_OBJECT} -> {status}").into());
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||
}
|
||||
}
|
||||
|
||||
async fn assert_anonymous_denied(response: reqwest::Response, what: &str) -> TestResult {
|
||||
let status = response.status().as_u16();
|
||||
let body = response.text().await?;
|
||||
assert_eq!(status, 403, "{what} must be denied, got {status}: {body}");
|
||||
assert!(body.contains("AccessDenied"), "{what} must carry AccessDenied: {body}");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Enforcement on: a public bucket policy does not exempt anonymous requests
|
||||
/// from per-key authorization, on either the read or the default-encryption
|
||||
/// write path.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn anonymous_sse_kms_denied_under_enforcement() -> TestResult {
|
||||
init_logging();
|
||||
|
||||
let mut env = LocalKMSTestEnvironment::new().await?;
|
||||
start_public_sse_kms_bucket(&mut env, true).await?;
|
||||
wait_for_public_read(&env).await?;
|
||||
|
||||
let read = anonymous_get(&env, ENCRYPTED_OBJECT).await?;
|
||||
assert_anonymous_denied(read, "anonymous GET of an SSE-KMS object").await?;
|
||||
|
||||
let write = anonymous_put(&env, "anon-write.txt").await?;
|
||||
assert_anonymous_denied(write, "anonymous PUT through an SSE-KMS default-encryption rule").await?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Enforcement off (the default): bucket policy alone governs anonymous access,
|
||||
/// and the default-encryption rule still encrypts anonymous writes.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn anonymous_sse_kms_governed_by_bucket_policy_without_enforcement() -> TestResult {
|
||||
init_logging();
|
||||
|
||||
let mut env = LocalKMSTestEnvironment::new().await?;
|
||||
start_public_sse_kms_bucket(&mut env, false).await?;
|
||||
wait_for_public_read(&env).await?;
|
||||
|
||||
let read = anonymous_get(&env, ENCRYPTED_OBJECT).await?;
|
||||
assert_eq!(read.status().as_u16(), 200, "anonymous GET of a public SSE-KMS object must succeed");
|
||||
assert_eq!(read.bytes().await?.as_ref(), PAYLOAD, "the object must be served decrypted");
|
||||
|
||||
let write = anonymous_put(&env, "anon-write.txt").await?;
|
||||
assert_eq!(write.status().as_u16(), 200, "anonymous PUT to a public bucket must succeed");
|
||||
|
||||
let stored = env
|
||||
.base_env
|
||||
.create_s3_client()
|
||||
.head_object()
|
||||
.bucket(BUCKET)
|
||||
.key("anon-write.txt")
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(
|
||||
stored.server_side_encryption(),
|
||||
Some(&ServerSideEncryption::AwsKms),
|
||||
"the anonymous write must be encrypted by the bucket default rule"
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -57,9 +57,6 @@ mod copy_object_version_restore_sse_test;
|
||||
#[cfg(test)]
|
||||
mod configured_roundtrip_test;
|
||||
|
||||
#[cfg(test)]
|
||||
mod kms_anonymous_enforcement_test;
|
||||
|
||||
#[cfg(test)]
|
||||
mod kms_authorization_negative_matrix_test;
|
||||
|
||||
|
||||
@@ -218,6 +218,9 @@ faster-hex = { workspace = true }
|
||||
ratelimit = { workspace = true }
|
||||
aws-smithy-http-client = { workspace = true, default-features = false, features = ["rustls-aws-lc"] }
|
||||
|
||||
# High-performance hashing
|
||||
ahash = { workspace = true, features = ["serde"] }
|
||||
|
||||
# Observability and Metrics
|
||||
metrics = { workspace = true }
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use ahash::AHashMap;
|
||||
use super::{metadata_boundary, object_lock_boundary, runtime_boundary as runtime_sources};
|
||||
use crate::bucket::lifecycle::bucket_lifecycle_audit::{
|
||||
LcAuditEvent, LcEventSrc, emit_non_transitioned_expiration_event, emit_transition_complete_event,
|
||||
@@ -2870,7 +2871,7 @@ fn spawn_transition_transaction_recovery_once(api: Arc<ECStore>) {
|
||||
struct StaleMultipartUploadCandidate {
|
||||
path: String,
|
||||
initiated: OffsetDateTime,
|
||||
metadata: Option<HashMap<String, String>>,
|
||||
metadata: Option<AHashMap<String, String>>,
|
||||
}
|
||||
|
||||
fn parse_stale_uploads_duration(env_key: &str, default: StdDuration) -> StdDuration {
|
||||
@@ -2915,9 +2916,9 @@ async fn stale_upload_current_size(set: &Arc<SetDisks>, metadata: &HashMap<Strin
|
||||
stale_upload_current_size_with_opts(set, metadata, upload_dir, false).await
|
||||
}
|
||||
|
||||
async fn stale_upload_current_size_with_opts(
|
||||
async fn stale_upload_current_size_with_opts<S: std::hash::BuildHasher>(
|
||||
set: &Arc<SetDisks>,
|
||||
metadata: &HashMap<String, String>,
|
||||
metadata: &HashMap<String, String, S>,
|
||||
upload_dir: &str,
|
||||
no_lock: bool,
|
||||
) -> Option<usize> {
|
||||
@@ -2950,9 +2951,9 @@ async fn stale_upload_current_size_with_opts(
|
||||
)
|
||||
}
|
||||
|
||||
async fn stale_upload_lifecycle_due(
|
||||
async fn stale_upload_lifecycle_due<S: std::hash::BuildHasher>(
|
||||
set: &Arc<SetDisks>,
|
||||
metadata: &HashMap<String, String>,
|
||||
metadata: &HashMap<String, String, S>,
|
||||
initiated: OffsetDateTime,
|
||||
upload_dir: &str,
|
||||
no_lock: bool,
|
||||
@@ -2978,7 +2979,7 @@ async fn stale_upload_lifecycle_due(
|
||||
.unwrap_or_default(),
|
||||
is_latest: true,
|
||||
delete_marker: false,
|
||||
user_defined: metadata.clone(),
|
||||
user_defined: metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
|
||||
@@ -27,7 +27,7 @@ use rustfs_protos::{
|
||||
ConnectionEvictionLogLevel, evict_failed_connection_with_log_level, models::PingBodyBuilder,
|
||||
proto_gen::node_service::node_service_client::NodeServiceClient,
|
||||
};
|
||||
use std::{sync::OnceLock, time::Duration};
|
||||
use std::time::Duration;
|
||||
use tokio::time::timeout;
|
||||
use tonic::Request;
|
||||
use tonic::service::interceptor::InterceptedService;
|
||||
@@ -44,35 +44,11 @@ pub struct RemoteClient {
|
||||
}
|
||||
|
||||
impl RemoteClient {
|
||||
const ONLINE_CHECK_RESOURCE: &'static str = "health-lock-online";
|
||||
|
||||
pub fn new(endpoint: String) -> Self {
|
||||
Self { addr: endpoint }
|
||||
}
|
||||
|
||||
fn ping_body() -> Bytes {
|
||||
static BODY: OnceLock<Bytes> = OnceLock::new();
|
||||
BODY.get_or_init(|| {
|
||||
let mut fbb = flatbuffers::FlatBufferBuilder::new();
|
||||
let payload = fbb.create_vector(b"health-check");
|
||||
let mut builder = PingBodyBuilder::new(&mut fbb);
|
||||
builder.add_payload(payload);
|
||||
let root = builder.finish();
|
||||
fbb.finish(root, None);
|
||||
Bytes::copy_from_slice(fbb.finished_data())
|
||||
})
|
||||
.clone()
|
||||
}
|
||||
|
||||
fn build_ping_request() -> PingRequest {
|
||||
PingRequest {
|
||||
version: 1,
|
||||
body: Self::ping_body(),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn build_fresh_ping_request_for_test() -> PingRequest {
|
||||
let mut fbb = flatbuffers::FlatBufferBuilder::new();
|
||||
let payload = fbb.create_vector(b"health-check");
|
||||
let mut builder = PingBodyBuilder::new(&mut fbb);
|
||||
@@ -188,16 +164,6 @@ impl RemoteClient {
|
||||
)
|
||||
}
|
||||
|
||||
fn online_check_timeout() -> Duration {
|
||||
Duration::from_millis(
|
||||
rustfs_utils::get_env_u64(
|
||||
rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS,
|
||||
rustfs_config::DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS,
|
||||
)
|
||||
.max(1),
|
||||
)
|
||||
}
|
||||
|
||||
async fn execute_rpc<T, F>(&self, op: &'static str, resource_summary: &str, future: F) -> std::result::Result<T, LockError>
|
||||
where
|
||||
F: std::future::Future<Output = std::result::Result<T, tonic::Status>>,
|
||||
@@ -581,37 +547,24 @@ impl LockClient for RemoteClient {
|
||||
}
|
||||
|
||||
async fn is_online(&self) -> bool {
|
||||
let online_timeout = Self::online_check_timeout();
|
||||
match timeout(online_timeout, async {
|
||||
let mut client = self.get_client().await?;
|
||||
let ping_req = Request::new(Self::build_ping_request());
|
||||
self.execute_rpc("ping", Self::ONLINE_CHECK_RESOURCE, client.ping(ping_req))
|
||||
.await?;
|
||||
Ok::<(), LockError>(())
|
||||
})
|
||||
.await
|
||||
{
|
||||
Ok(Ok(())) => {
|
||||
debug!(addr = %self.addr, timeout_ms = online_timeout.as_millis(), "remote lock client is online");
|
||||
// Use Ping interface to test if remote service is online
|
||||
let mut client = match self.get_client().await {
|
||||
Ok(client) => client,
|
||||
Err(_) => {
|
||||
info!("remote client {} connection failed", self.addr);
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
let ping_req = Request::new(Self::build_ping_request());
|
||||
|
||||
match client.ping(ping_req).await {
|
||||
Ok(_) => {
|
||||
info!("remote client {} is online", self.addr);
|
||||
true
|
||||
}
|
||||
Ok(Err(err)) => {
|
||||
debug!(
|
||||
addr = %self.addr,
|
||||
timeout_ms = online_timeout.as_millis(),
|
||||
error = %err,
|
||||
"remote lock client online check failed"
|
||||
);
|
||||
false
|
||||
}
|
||||
Err(_) => {
|
||||
let reason = format!("online check timed out after {:?}", online_timeout);
|
||||
warn!(
|
||||
addr = %self.addr,
|
||||
timeout_ms = online_timeout.as_millis(),
|
||||
"remote lock client online check timed out"
|
||||
);
|
||||
self.evict_connection("ping", &reason, Self::ONLINE_CHECK_RESOURCE).await;
|
||||
info!("remote client {} ping failed", self.addr);
|
||||
false
|
||||
}
|
||||
}
|
||||
@@ -698,15 +651,6 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cached_ping_request_matches_fresh_flatbuffer_payload() {
|
||||
let cached = RemoteClient::build_ping_request();
|
||||
let fresh = RemoteClient::build_fresh_ping_request_for_test();
|
||||
|
||||
assert_eq!(cached.version, fresh.version);
|
||||
assert_eq!(cached.body, fresh.body);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn test_remote_client_acquire_lock_uses_rpc_timeout_and_evicts_connection() {
|
||||
@@ -835,48 +779,6 @@ mod tests {
|
||||
accept_task.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn test_remote_client_is_online_uses_health_timeout_and_evicts_connection() {
|
||||
ensure_test_rpc_secret();
|
||||
let Some((addr, accept_task)) = spawn_hanging_listener().await else {
|
||||
return;
|
||||
};
|
||||
cache_lazy_channel(&addr).await;
|
||||
assert!(runtime_sources::test_node_channel_is_cached(&addr).await);
|
||||
|
||||
temp_env::async_with_vars(
|
||||
[
|
||||
(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("50")),
|
||||
(rustfs_config::ENV_OBJECT_LOCK_RPC_TIMEOUT_MS, Some("1000")),
|
||||
],
|
||||
async {
|
||||
let client = RemoteClient::new(addr.clone());
|
||||
let started_at = tokio::time::Instant::now();
|
||||
|
||||
let online = client.is_online().await;
|
||||
let elapsed = started_at.elapsed();
|
||||
|
||||
assert!(!online, "hanging remote lock peer must not be reported online");
|
||||
assert!(
|
||||
elapsed >= Duration::from_millis(40),
|
||||
"remote online check should honor configured health timeout, got {elapsed:?}"
|
||||
);
|
||||
assert!(
|
||||
elapsed < Duration::from_secs(1),
|
||||
"health timeout should keep readiness probes bounded, got {elapsed:?}"
|
||||
);
|
||||
assert!(
|
||||
!runtime_sources::test_node_channel_is_cached(&addr).await,
|
||||
"online-check timeout should evict cached connection"
|
||||
);
|
||||
},
|
||||
)
|
||||
.await;
|
||||
|
||||
accept_task.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn test_remote_client_refresh_tonic_error_evicts_connection() {
|
||||
@@ -1004,21 +906,4 @@ mod tests {
|
||||
assert_eq!(RemoteClient::rpc_timeout(), Duration::from_millis(1));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial]
|
||||
fn test_remote_client_online_timeout_honors_configured_deadline() {
|
||||
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, None::<&str>, || {
|
||||
assert_eq!(
|
||||
RemoteClient::online_check_timeout(),
|
||||
Duration::from_millis(rustfs_config::DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS)
|
||||
);
|
||||
});
|
||||
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("50"), || {
|
||||
assert_eq!(RemoteClient::online_check_timeout(), Duration::from_millis(50));
|
||||
});
|
||||
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("0"), || {
|
||||
assert_eq!(RemoteClient::online_check_timeout(), Duration::from_millis(1));
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2793,13 +2793,12 @@ where
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{
|
||||
SERVER_CONFIG_LOCK, ServerConfigSnapshot, apply_dynamic_config_for_sub_sys_with, build_scalar_config_object,
|
||||
config_task_join_error, configs_semantically_equal, decode_server_config_blob, encode_server_config_blob,
|
||||
heal_config_descriptor, is_standard_object_server_config, lookup_configs, new_and_save_server_config, read_config,
|
||||
read_config_no_lock_preserve_empty_with_metadata, read_config_preserve_empty, read_config_with_metadata,
|
||||
read_config_without_migrate, read_server_config_snapshot, save_server_config, save_server_config_snapshot,
|
||||
save_server_config_snapshot_with_generation, server_config_transaction_lock_path, should_warn_ignored_scalar_section,
|
||||
storage_class_kvs_mut,
|
||||
SERVER_CONFIG_LOCK, ServerConfigSnapshot, apply_dynamic_config_for_sub_sys_with, config_task_join_error,
|
||||
configs_semantically_equal, decode_server_config_blob, encode_server_config_blob, is_standard_object_server_config,
|
||||
lookup_configs, new_and_save_server_config, read_config, read_config_no_lock_preserve_empty_with_metadata,
|
||||
read_config_preserve_empty, read_config_with_metadata, read_config_without_migrate, read_server_config_snapshot,
|
||||
save_server_config, save_server_config_snapshot, save_server_config_snapshot_with_generation,
|
||||
server_config_transaction_lock_path, should_warn_ignored_scalar_section, storage_class_kvs_mut,
|
||||
};
|
||||
use crate::config::{audit, heal, notify, oidc, scanner};
|
||||
use crate::disk::endpoint::Endpoint;
|
||||
@@ -3542,31 +3541,6 @@ mod tests {
|
||||
cfg
|
||||
}
|
||||
|
||||
/// `Config::new()` (and every decode built on it) reads the process-global
|
||||
/// `rustfs_config::server_config::DEFAULT_KVS` OnceLock at call time, and
|
||||
/// other tests in this binary register it via `crate::config::init()`
|
||||
/// mid-run. Equality assertions must therefore normalize both sides with
|
||||
/// one snapshot taken after both configs exist, never against a later
|
||||
/// `Config::new()`.
|
||||
fn default_kvs_snapshot() -> Option<&'static std::collections::HashMap<String, KVS>> {
|
||||
rustfs_config::server_config::DEFAULT_KVS.get()
|
||||
}
|
||||
|
||||
/// Fills the default sections `cfg` is missing from an explicit
|
||||
/// [`default_kvs_snapshot`], mirroring `Config::set_defaults`.
|
||||
fn filled_with_default_kvs(mut cfg: Config, snapshot: Option<&std::collections::HashMap<String, KVS>>) -> Config {
|
||||
if let Some(defaults) = snapshot {
|
||||
for (sub_sys, kvs) in defaults {
|
||||
cfg.0
|
||||
.entry(sub_sys.clone())
|
||||
.or_default()
|
||||
.entry(DEFAULT_DELIMITER.to_string())
|
||||
.or_insert_with(|| kvs.clone());
|
||||
}
|
||||
}
|
||||
cfg
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_external_scanner_config_decodes_with_defaults() {
|
||||
let cfg =
|
||||
@@ -3656,9 +3630,7 @@ mod tests {
|
||||
}"#;
|
||||
|
||||
let cfg = decode_server_config_blob(seed).expect("root heal null should mean no persisted override");
|
||||
// The heal section may hold registered defaults, so assert on the
|
||||
// semantic diff instead of the section's presence.
|
||||
assert!(build_scalar_config_object(&cfg, heal_config_descriptor()).is_empty());
|
||||
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
|
||||
assert!(!is_standard_object_server_config(seed));
|
||||
|
||||
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("legacy seed should canonicalize on an authorized save");
|
||||
@@ -3693,12 +3665,7 @@ mod tests {
|
||||
let input = format!(r#"{{"version":"33","storageclass":{{"standard":"","rrs":""}},{section}}}"#);
|
||||
let cfg = decode_server_config_blob(input.as_bytes())
|
||||
.unwrap_or_else(|err| panic!("legacy scalar section {section} should be ignored, got: {err}"));
|
||||
let snapshot = default_kvs_snapshot();
|
||||
assert_eq!(
|
||||
filled_with_default_kvs(cfg.clone(), snapshot),
|
||||
filled_with_default_kvs(base.clone(), snapshot),
|
||||
"ignored section {section} must contribute no overrides"
|
||||
);
|
||||
assert_eq!(cfg, base, "ignored section {section} must contribute no overrides");
|
||||
assert!(
|
||||
!is_standard_object_server_config(input.as_bytes()),
|
||||
"seed with {section} must not count as standard so a save rewrites it"
|
||||
@@ -3776,19 +3743,12 @@ mod tests {
|
||||
fn valid_heal_object_and_kvs_array_shapes_remain_accepted() {
|
||||
let empty_object = br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":{}}"#;
|
||||
let cfg = decode_server_config_blob(empty_object).expect("empty heal object should decode as no override");
|
||||
// The heal section may hold registered defaults, so assert on the
|
||||
// semantic diff instead of the section's presence.
|
||||
assert!(build_scalar_config_object(&cfg, heal_config_descriptor()).is_empty());
|
||||
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
|
||||
|
||||
let kvs_array =
|
||||
br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":[{"key":"bitrot_cycle","value":"off"}]}"#;
|
||||
let cfg = decode_server_config_blob(kvs_array).expect("heal KVS array should decode");
|
||||
assert_eq!(
|
||||
build_scalar_config_object(&cfg, heal_config_descriptor())
|
||||
.get(HEAL_BITROT_CYCLE)
|
||||
.and_then(Value::as_str),
|
||||
Some("off")
|
||||
);
|
||||
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -4940,12 +4900,8 @@ mod tests {
|
||||
fn test_fallback_returns_default_config_when_recovery_enabled() {
|
||||
let cfg = fallback_server_config_after_corruption(corrupt_config_error(), "config/config.json", true)
|
||||
.expect("recovery enabled must fall back to the default config");
|
||||
let snapshot = default_kvs_snapshot();
|
||||
assert!(
|
||||
configs_semantically_equal(
|
||||
&filled_with_default_kvs(cfg, snapshot),
|
||||
&filled_with_default_kvs(Config(std::collections::HashMap::new()), snapshot)
|
||||
),
|
||||
configs_semantically_equal(&cfg, &Config::new()),
|
||||
"fallback config should be the default server config"
|
||||
);
|
||||
}
|
||||
@@ -5562,12 +5518,8 @@ mod tests {
|
||||
.expect("unrecoverable corruption should fall back to the default config");
|
||||
|
||||
assert_eq!(store.heal_calls.load(Ordering::SeqCst), 1, "heal should be attempted before falling back");
|
||||
let snapshot = default_kvs_snapshot();
|
||||
assert!(
|
||||
configs_semantically_equal(
|
||||
&filled_with_default_kvs(cfg, snapshot),
|
||||
&filled_with_default_kvs(Config(std::collections::HashMap::new()), snapshot)
|
||||
),
|
||||
configs_semantically_equal(&cfg, &Config::new()),
|
||||
"fallback config should be the default server config"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -96,8 +96,6 @@ const LOG_SUBSYSTEM_POOLS: &str = "pools";
|
||||
const EVENT_DECOMMISSION_STATE: &str = "decommission_state";
|
||||
const EVENT_DECOMMISSION_BUCKET: &str = "decommission_bucket";
|
||||
const EVENT_DECOMMISSION_ENTRY: &str = "decommission_entry";
|
||||
const POOL_ACTIVATION_FLEET_PROOF_REQUIRED: &str = "pool activation requires a live fleet capability proof";
|
||||
const POOL_ACTIVATION_FLEET_PROOF_EXPIRED: &str = "pool activation fleet capability proof expired before commit";
|
||||
const DECOMMISSION_STAGE_MIGRATE_OBJECT: &str = "migrate_object";
|
||||
const DECOMMISSION_STAGE_CLEANUP_PREFLIGHT: &str = "cleanup_preflight";
|
||||
const DECOMMISSION_STAGE_SOURCE_CLEANUP: &str = "source_cleanup";
|
||||
@@ -1834,13 +1832,6 @@ pub(crate) struct PoolRebalanceActivationFence {
|
||||
}
|
||||
|
||||
impl PoolRebalanceActivationFence {
|
||||
pub(crate) fn set_fleet_proof(
|
||||
&mut self,
|
||||
fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
|
||||
) {
|
||||
self.fleet_proof = fleet_proof;
|
||||
}
|
||||
|
||||
pub(crate) fn ensure_held(&self) -> Result<()> {
|
||||
#[cfg(test)]
|
||||
let forced_lost = self.forced_lost.load(Ordering::Acquire);
|
||||
@@ -1854,7 +1845,7 @@ impl PoolRebalanceActivationFence {
|
||||
.as_ref()
|
||||
.is_some_and(|proof| !crate::services::notification_sys::cross_pool_fence_fleet_proof_matches(proof))
|
||||
{
|
||||
return Err(Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED));
|
||||
return Err(Error::other("pool activation fleet capability proof expired before commit"));
|
||||
}
|
||||
|
||||
Ok(())
|
||||
@@ -1910,17 +1901,7 @@ pub(crate) async fn acquire_pool_activation_fleet_proof(
|
||||
}
|
||||
crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof()
|
||||
.map(Some)
|
||||
.ok_or_else(|| Error::other(POOL_ACTIVATION_FLEET_PROOF_REQUIRED))
|
||||
}
|
||||
|
||||
pub(crate) fn is_pool_activation_fleet_proof_error(err: &Error) -> bool {
|
||||
// Save-stage helpers add context by formatting the original error, so the
|
||||
// marker may be nested in the display string. Restrict matching to the
|
||||
// `Error::other` I/O shape used by this activation path.
|
||||
matches!(err, Error::Io(io_error) if io_error.kind() == std::io::ErrorKind::Other && {
|
||||
let message = io_error.to_string();
|
||||
message.contains(POOL_ACTIVATION_FLEET_PROOF_REQUIRED) || message.contains(POOL_ACTIVATION_FLEET_PROOF_EXPIRED)
|
||||
})
|
||||
.ok_or_else(|| Error::other("pool activation requires a live fleet capability proof"))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -5815,7 +5796,7 @@ fn decommission_remote_tiered_opts(
|
||||
versioned: version_id.is_some(),
|
||||
version_id,
|
||||
mod_time: version.mod_time,
|
||||
user_defined: version.metadata.clone(),
|
||||
user_defined: version.metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect(),
|
||||
src_pool_idx,
|
||||
data_movement: true,
|
||||
incl_free_versions: version.tier_free_version(),
|
||||
@@ -10847,15 +10828,6 @@ mod tests {
|
||||
use crate::bucket::replication::{ReplicationState, ReplicationStatusType};
|
||||
use serde::Serialize;
|
||||
|
||||
#[test]
|
||||
fn pool_activation_fleet_proof_error_classifier_matches_only_retryable_proof_failures() {
|
||||
assert!(is_pool_activation_fleet_proof_error(&Error::other(POOL_ACTIVATION_FLEET_PROOF_REQUIRED)));
|
||||
assert!(is_pool_activation_fleet_proof_error(&Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED)));
|
||||
let wrapped = format!("rebalance meta save failed during start_rebalance: {POOL_ACTIVATION_FLEET_PROOF_EXPIRED}");
|
||||
assert!(is_pool_activation_fleet_proof_error(&Error::other(wrapped)));
|
||||
assert!(!is_pool_activation_fleet_proof_error(&Error::ConfigNotFound));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn decommission_activation_fence_loss_after_durable_save_blocks_publication() {
|
||||
|
||||
@@ -157,8 +157,6 @@ pub enum StorageError {
|
||||
InvalidPartNumber(usize),
|
||||
#[error("Your proposed upload is smaller than the minimum allowed size. Part {0} size {1} is less than minimum {2}")]
|
||||
EntityTooSmall(usize, i64, i64),
|
||||
#[error("multipart upload size {0} exceeds the configured limit {1}")]
|
||||
EntityTooLarge(u64, u64),
|
||||
|
||||
// ── Erasure / Quorum ─────────────────────────────────────────────
|
||||
#[error("erasure read quorum")]
|
||||
@@ -556,7 +554,6 @@ impl Clone for StorageError {
|
||||
StorageError::DecommissionNotStarted => StorageError::DecommissionNotStarted,
|
||||
StorageError::InvalidPart(a, b, c) => StorageError::InvalidPart(*a, b.clone(), c.clone()),
|
||||
StorageError::EntityTooSmall(a, b, c) => StorageError::EntityTooSmall(*a, *b, *c),
|
||||
StorageError::EntityTooLarge(a, b) => StorageError::EntityTooLarge(*a, *b),
|
||||
StorageError::DoneForNow => StorageError::DoneForNow,
|
||||
StorageError::DecommissionAlreadyRunning => StorageError::DecommissionAlreadyRunning,
|
||||
StorageError::RebalanceAlreadyRunning => StorageError::RebalanceAlreadyRunning,
|
||||
@@ -676,7 +673,6 @@ impl StorageError {
|
||||
StorageError::InsufficientWriteQuorum(_, _) => StorageErrorCode::InsufficientWriteQuorum,
|
||||
StorageError::PreconditionFailed => StorageErrorCode::PreconditionFailed,
|
||||
StorageError::EntityTooSmall(_, _, _) => StorageErrorCode::EntityTooSmall,
|
||||
StorageError::EntityTooLarge(_, _) => StorageErrorCode::EntityTooLarge,
|
||||
StorageError::InvalidRangeSpec(_) => StorageErrorCode::InvalidRangeSpec,
|
||||
StorageError::NotModified => StorageErrorCode::NotModified,
|
||||
StorageError::InvalidPartNumber(_) => StorageErrorCode::InvalidPartNumber,
|
||||
@@ -799,7 +795,6 @@ impl StorageError {
|
||||
StorageErrorCode::EntityTooSmall => {
|
||||
Some(StorageError::EntityTooSmall(Default::default(), Default::default(), Default::default()))
|
||||
}
|
||||
StorageErrorCode::EntityTooLarge => Some(StorageError::EntityTooLarge(Default::default(), Default::default())),
|
||||
StorageErrorCode::InvalidRangeSpec => Some(StorageError::InvalidRangeSpec(Default::default())),
|
||||
StorageErrorCode::NotModified => Some(StorageError::NotModified),
|
||||
StorageErrorCode::InvalidPartNumber => Some(StorageError::InvalidPartNumber(Default::default())),
|
||||
|
||||
@@ -64,7 +64,7 @@ pub(crate) const ENCRYPTED_FRAME_LAYOUT_FIXED8K_SUFFIX: &str = "encrypted-frame-
|
||||
pub(crate) const ENV_RUSTFS_ENCRYPTED_RANGE_SEEK: &str = "RUSTFS_ENCRYPTED_RANGE_SEEK";
|
||||
pub(crate) const DEFAULT_RUSTFS_ENCRYPTED_RANGE_SEEK: bool = true;
|
||||
|
||||
pub(crate) fn has_encrypted_part_layout_marker(metadata: &HashMap<String, String>, suffix: &str, expected: &str) -> bool {
|
||||
pub(crate) fn has_encrypted_part_layout_marker<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>, suffix: &str, expected: &str) -> bool {
|
||||
let mut value = None;
|
||||
for (key, candidate) in metadata {
|
||||
if !rustfs_utils::http::has_internal_suffix(key, suffix) {
|
||||
|
||||
@@ -168,7 +168,7 @@ pub fn to_s3s_etag(etag: &str) -> ETag {
|
||||
ETag::Strong(etag.to_string())
|
||||
}
|
||||
|
||||
pub fn get_raw_etag(metadata: &HashMap<String, String>) -> String {
|
||||
pub fn get_raw_etag<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> String {
|
||||
metadata
|
||||
.get("etag")
|
||||
.cloned()
|
||||
|
||||
@@ -1008,7 +1008,7 @@ impl ObjectInfo {
|
||||
successor_mod_time: fi.successor_mod_time,
|
||||
etag,
|
||||
inlined,
|
||||
user_defined: Arc::new(metadata),
|
||||
user_defined: Arc::new(metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect()),
|
||||
transitioned_object,
|
||||
transition_version_state: fi.transition_version_state,
|
||||
checksum: fi.checksum.clone(),
|
||||
@@ -1316,7 +1316,7 @@ impl ObjectInfo {
|
||||
if part > 0
|
||||
&& let Some(checksums) = self.parts.iter().find(|p| p.number == part).and_then(|p| p.checksums.clone())
|
||||
{
|
||||
return Ok((checksums, true));
|
||||
return Ok((checksums.iter().map(|(k, v)| (k.clone(), v.clone())).collect(), true));
|
||||
}
|
||||
|
||||
if let Some(data) = &self.checksum {
|
||||
|
||||
@@ -234,39 +234,6 @@ pub(crate) fn install_cross_pool_fence_fleet_proof_for_test() {
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) struct CrossPoolFenceFleetProofGuard {
|
||||
previous_proof: Option<FleetCapabilityProof>,
|
||||
previous_topology_conflict: bool,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
impl Drop for CrossPoolFenceFleetProofGuard {
|
||||
fn drop(&mut self) {
|
||||
let mut state = cross_pool_fence_fleet_proof_slot()
|
||||
.write()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
state.proof = self.previous_proof.take();
|
||||
state.topology_conflict = self.previous_topology_conflict;
|
||||
}
|
||||
}
|
||||
|
||||
/// Temporarily revoke the test proof so activation paths can exercise their
|
||||
/// fail-closed behavior without changing the process-wide topology binding.
|
||||
#[cfg(test)]
|
||||
pub(crate) fn without_cross_pool_fence_fleet_proof_for_test() -> CrossPoolFenceFleetProofGuard {
|
||||
let mut state = cross_pool_fence_fleet_proof_slot()
|
||||
.write()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
let guard = CrossPoolFenceFleetProofGuard {
|
||||
previous_proof: state.proof.clone(),
|
||||
previous_topology_conflict: state.topology_conflict,
|
||||
};
|
||||
state.proof = None;
|
||||
state.topology_conflict = true;
|
||||
guard
|
||||
}
|
||||
|
||||
#[cfg(any(test, feature = "test-util"))]
|
||||
pub fn rotate_cross_pool_fence_fleet_proof_for_test() -> bool {
|
||||
let mut state = cross_pool_fence_fleet_proof_slot()
|
||||
|
||||
@@ -570,13 +570,10 @@ impl ECStore {
|
||||
where
|
||||
S: EcstoreObjectIO + StorageNamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
|
||||
{
|
||||
// Lock order: pool_meta_save_gate -> pool.bin -> rebalance.bin.
|
||||
let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
|
||||
let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
|
||||
pool_meta_guard.ensure_write_safe("rebalance worker activation")?;
|
||||
// Classify the durable rebalance record while holding both namespace
|
||||
// fences. A terminal record is a no-op and must not depend on the
|
||||
// notification subsystem having published a fleet proof yet.
|
||||
let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?;
|
||||
let activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), fleet_proof).await?;
|
||||
let pool_meta = self
|
||||
.load_runtime_pool_meta_under_activation_fence(&mut pool_meta_guard, &activation_fence, "rebalance worker activation")
|
||||
.await?;
|
||||
@@ -600,17 +597,10 @@ impl ECStore {
|
||||
}
|
||||
|
||||
activation_fence.ensure_held()?;
|
||||
if !crate::services::rebalance::rebalance_requires_worker_activation(&persisted) {
|
||||
if !is_rebalance_conflicting_with_decommission(&persisted) {
|
||||
return Ok(RebalanceWorkerActivationFence::NotStartedTerminal);
|
||||
}
|
||||
|
||||
// Active worker admission still requires the fail-closed fleet proof.
|
||||
// Attach it immediately before the final fence validation so expiry or
|
||||
// topology changes are checked again at every later commit boundary.
|
||||
let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
|
||||
activation_fence.set_fleet_proof(fleet_proof);
|
||||
activation_fence.ensure_held()?;
|
||||
|
||||
Ok(RebalanceWorkerActivationFence::Ready(Box::new(activation_fence)))
|
||||
}
|
||||
|
||||
@@ -1486,64 +1476,6 @@ mod tests {
|
||||
assert_activation_locks_released(&store).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn rebalance_worker_skips_terminal_metadata_without_fleet_proof() {
|
||||
let rebalance_id = "terminal-metadata-without-proof";
|
||||
let completed = RebalanceMeta {
|
||||
id: rebalance_id.to_string(),
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Completed,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(Some(completed)).await;
|
||||
let _proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
|
||||
|
||||
let activation = store
|
||||
.fence_rebalance_worker_activation(store.pools[0].clone(), rebalance_id)
|
||||
.await
|
||||
.expect("terminal metadata should not require a fleet proof");
|
||||
assert!(matches!(activation, RebalanceWorkerActivationFence::NotStartedTerminal));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn rebalance_worker_still_requires_fleet_proof_for_active_metadata() {
|
||||
let rebalance_id = "active-metadata-without-proof";
|
||||
let active = RebalanceMeta {
|
||||
id: rebalance_id.to_string(),
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Started,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(Some(active)).await;
|
||||
let _proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
|
||||
|
||||
let err = match store
|
||||
.fence_rebalance_worker_activation(store.pools[0].clone(), rebalance_id)
|
||||
.await
|
||||
{
|
||||
Ok(_) => panic!("active metadata must not be admitted without a fleet proof"),
|
||||
Err(err) => err,
|
||||
};
|
||||
assert!(
|
||||
err.to_string()
|
||||
.contains("pool activation requires a live fleet capability proof")
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn rebalance_activation_adopts_commit_after_post_save_fence_loss() {
|
||||
|
||||
@@ -214,14 +214,6 @@ pub(super) fn is_rebalance_in_progress(meta: &RebalanceMeta) -> bool {
|
||||
meta.pool_stats.iter().any(is_rebalance_pool_active)
|
||||
}
|
||||
|
||||
/// Persisted rebalance metadata requires worker activation only while it has
|
||||
/// not reached a durable terminal marker and at least one pool is still marked
|
||||
/// active. Merely finding `rebalance.bin` is not evidence that admission is
|
||||
/// required: terminal metadata is retained for status reporting.
|
||||
pub(crate) fn rebalance_requires_worker_activation(meta: &RebalanceMeta) -> bool {
|
||||
meta.stopped_at.is_none() && is_rebalance_in_progress(meta)
|
||||
}
|
||||
|
||||
pub(crate) fn is_rebalance_conflicting_with_decommission(meta: &RebalanceMeta) -> bool {
|
||||
is_rebalance_in_progress(meta)
|
||||
}
|
||||
|
||||
@@ -57,7 +57,7 @@ fn rebalance_remote_tiered_opts(
|
||||
versioned: version_id.is_some(),
|
||||
version_id,
|
||||
mod_time: version.mod_time,
|
||||
user_defined: version.metadata.clone(),
|
||||
user_defined: version.metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect(),
|
||||
src_pool_idx,
|
||||
data_movement: true,
|
||||
include_part_checksums: true,
|
||||
|
||||
@@ -49,8 +49,8 @@ mod worker;
|
||||
|
||||
#[cfg(feature = "test-util")]
|
||||
pub use entry::test_util::PausedRebalanceEntryTestFixture;
|
||||
pub(crate) use meta::is_rebalance_conflicting_with_decommission;
|
||||
pub use meta::{decode_rebalance_stop_propagation_record, encode_rebalance_stop_propagation_record};
|
||||
pub(crate) use meta::{is_rebalance_conflicting_with_decommission, rebalance_requires_worker_activation};
|
||||
pub use types::{
|
||||
DiskStat, RebalSaveOpt, RebalStatus, RebalanceCleanupWarningEntry, RebalanceCleanupWarnings, RebalanceInfo, RebalanceMeta,
|
||||
RebalanceStats, RebalanceStopPropagationRecord,
|
||||
|
||||
@@ -22,11 +22,11 @@ use super::meta::{
|
||||
is_rebalance_in_progress, is_rebalance_meta_replaceable_for_new_id, is_rebalance_stopped_terminal_event,
|
||||
mark_rebalance_bucket_done, merge_rebalance_bucket_lists, merge_rebalance_meta, next_rebal_bucket_from_stat,
|
||||
percent_free_ratio, rebalance_goal_reached, rebalance_meta_load_no_data_error, rebalance_meta_load_unknown_format_error,
|
||||
rebalance_meta_load_unknown_version_error, rebalance_requires_worker_activation, record_rebalance_cleanup_warning_in_meta,
|
||||
remove_rebalanced_buckets_from_queue, resolve_next_rebalance_bucket, resolve_rebalance_participants,
|
||||
should_accept_rebalance_stats_update, should_ignore_rebalance_data_usage_cache, should_pool_participate,
|
||||
should_preserve_rebalance_stopped_state, should_skip_start_rebalance, stop_rebalance_meta_snapshot, stop_rebalance_state,
|
||||
take_bucket_from_rebalance_queue, validate_init_rebalance_state, validate_start_rebalance_state,
|
||||
rebalance_meta_load_unknown_version_error, record_rebalance_cleanup_warning_in_meta, remove_rebalanced_buckets_from_queue,
|
||||
resolve_next_rebalance_bucket, resolve_rebalance_participants, should_accept_rebalance_stats_update,
|
||||
should_ignore_rebalance_data_usage_cache, should_pool_participate, should_preserve_rebalance_stopped_state,
|
||||
should_skip_start_rebalance, stop_rebalance_meta_snapshot, stop_rebalance_state, take_bucket_from_rebalance_queue,
|
||||
validate_init_rebalance_state, validate_start_rebalance_state,
|
||||
};
|
||||
use super::migration::{
|
||||
MigrationBackend, MigrationVersionResult, migrate_entry_version, migrate_entry_version_with_retry_wait,
|
||||
@@ -3386,52 +3386,6 @@ fn test_is_rebalance_in_progress_only_started_participants() {
|
||||
assert!(is_rebalance_in_progress(&meta));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_rebalance_requires_worker_activation_only_for_active_non_stopped_metadata() {
|
||||
let now = OffsetDateTime::now_utc();
|
||||
let active = RebalanceMeta {
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Started,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
let stopped_active = RebalanceMeta {
|
||||
stopped_at: Some(now),
|
||||
pool_stats: active.pool_stats.clone(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
assert!(rebalance_requires_worker_activation(&active));
|
||||
for status in [
|
||||
RebalStatus::Completed,
|
||||
RebalStatus::Stopped,
|
||||
RebalStatus::Failed,
|
||||
RebalStatus::None,
|
||||
] {
|
||||
let terminal = RebalanceMeta {
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
assert!(
|
||||
!rebalance_requires_worker_activation(&terminal),
|
||||
"terminal status {status:?} must not resume"
|
||||
);
|
||||
}
|
||||
assert!(!rebalance_requires_worker_activation(&stopped_active));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_is_rebalance_conflicting_with_decommission_true_when_in_progress() {
|
||||
let now = OffsetDateTime::now_utc();
|
||||
|
||||
@@ -1454,7 +1454,7 @@ fn tier_backend_identity(config: &TierConfig) -> io::Result<TierDestinationId> {
|
||||
encode_tier_backend_identity(tier_type, endpoint, bucket, prefix, region, routing_account)
|
||||
}
|
||||
|
||||
pub(crate) fn tier_destination_id_from_metadata(metadata: &HashMap<String, String>) -> io::Result<Option<TierDestinationId>> {
|
||||
pub(crate) fn tier_destination_id_from_metadata<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> io::Result<Option<TierDestinationId>> {
|
||||
let Some(encoded) = rustfs_utils::http::metadata_compat::get_consistent_str(
|
||||
metadata,
|
||||
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID,
|
||||
|
||||
@@ -609,7 +609,7 @@ impl SetDisks {
|
||||
|| Self::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX)
|
||||
}
|
||||
|
||||
fn update_hash_quorum_metadata_map(hasher: &mut Sha256, entries: &HashMap<String, String>) {
|
||||
fn update_hash_quorum_metadata_map<S: std::hash::BuildHasher>(hasher: &mut Sha256, entries: &HashMap<String, String, S>) {
|
||||
let mut entries = entries
|
||||
.iter()
|
||||
.filter(|(name, _)| !Self::is_replication_quorum_metadata_key(name))
|
||||
@@ -635,7 +635,7 @@ impl SetDisks {
|
||||
/// so the dual internal prefixes carrying the same mapping share one
|
||||
/// identity, while a genuine disagreement between disks still changes the
|
||||
/// hash and surfaces as a quorum difference.
|
||||
fn update_hash_target_delete_marker_versions(hasher: &mut Sha256, metadata: &HashMap<String, String>) {
|
||||
fn update_hash_target_delete_marker_versions<S: std::hash::BuildHasher>(hasher: &mut Sha256, metadata: &HashMap<String, String, S>) {
|
||||
let (versions, corrupt) = http::target_delete_marker_versions(metadata);
|
||||
hasher.update([u8::from(corrupt)]);
|
||||
let mut versions = versions.iter().collect::<Vec<_>>();
|
||||
|
||||
@@ -190,7 +190,7 @@ use tracing::error;
|
||||
use tracing::{Instrument, debug, info, warn};
|
||||
use uuid::Uuid;
|
||||
|
||||
pub(super) fn restore_operation_id_from_metadata(metadata: &HashMap<String, String>) -> Result<Option<Uuid>> {
|
||||
pub(super) fn restore_operation_id_from_metadata<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> Result<Option<Uuid>> {
|
||||
let Some(value) = rustfs_utils::http::metadata_compat::get_consistent_str(metadata, SUFFIX_RESTORE_OPERATION_ID) else {
|
||||
if rustfs_utils::http::metadata_compat::contains_key_str(metadata, SUFFIX_RESTORE_OPERATION_ID) {
|
||||
return Err(Error::other("invalid restore operation id metadata".to_string()));
|
||||
@@ -204,14 +204,14 @@ pub(super) fn restore_operation_id_from_metadata(metadata: &HashMap<String, Stri
|
||||
Ok(Some(id))
|
||||
}
|
||||
|
||||
pub(super) fn require_restore_operation_id(metadata: &HashMap<String, String>, expected: Uuid) -> Result<()> {
|
||||
pub(super) fn require_restore_operation_id<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>, expected: Uuid) -> Result<()> {
|
||||
match restore_operation_id_from_metadata(metadata)? {
|
||||
Some(actual) if actual == expected => Ok(()),
|
||||
_ => Err(Error::other("restore operation id changed before copy-back".to_string())),
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn restore_commit_operation_id_from_metadata(metadata: &HashMap<String, String>) -> Result<Option<Uuid>> {
|
||||
pub(super) fn restore_commit_operation_id_from_metadata<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> Result<Option<Uuid>> {
|
||||
if !metadata.contains_key(X_AMZ_RESTORE.as_str()) {
|
||||
return Ok(None);
|
||||
}
|
||||
@@ -466,13 +466,13 @@ fn release_materialized_read_lock(bucket: &str, object: &str, read_lock_guard: O
|
||||
drop(read_lock_guard);
|
||||
}
|
||||
|
||||
pub(crate) fn strip_internal_multipart_metadata(metadata: &mut HashMap<String, String>) {
|
||||
pub(crate) fn strip_internal_multipart_metadata<S: std::hash::BuildHasher>(metadata: &mut HashMap<String, String, S>) {
|
||||
metadata.remove(RUSTFS_MULTIPART_BUCKET_KEY);
|
||||
metadata.remove(RUSTFS_MULTIPART_OBJECT_KEY);
|
||||
rustfs_utils::http::metadata_compat::remove_str(metadata, SUFFIX_BUCKET_INCARNATION_ID);
|
||||
}
|
||||
|
||||
fn should_persist_encryption_original_size(metadata: &HashMap<String, String>) -> bool {
|
||||
fn should_persist_encryption_original_size<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> bool {
|
||||
metadata.keys().any(|key| is_object_encryption_marker(key))
|
||||
}
|
||||
|
||||
|
||||
@@ -84,13 +84,11 @@ use rustfs_rio::TryGetIndex;
|
||||
use rustfs_utils::http::SSEC_ALGORITHM_HEADER;
|
||||
#[cfg(test)]
|
||||
use rustfs_utils::http::SUFFIX_COMPRESSION;
|
||||
use rustfs_utils::http::{SUFFIX_MAX_TOTAL_OBJECT_SIZE, get_consistent_str};
|
||||
use std::future::Future;
|
||||
#[cfg(test)]
|
||||
use std::sync::atomic::AtomicBool;
|
||||
#[cfg(any(test, feature = "test-util"))]
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::sync::{Mutex, OnceLock};
|
||||
#[cfg(any(test, feature = "test-util"))]
|
||||
use std::time::Duration;
|
||||
#[cfg(test)]
|
||||
@@ -99,83 +97,6 @@ use tokio::task::JoinSet;
|
||||
|
||||
const MULTIPART_LIST_IO_CONCURRENCY: usize = 16;
|
||||
|
||||
static CAPPED_MULTIPART_STAGING: OnceLock<Mutex<HashMap<String, Arc<tokio::sync::Semaphore>>>> = OnceLock::new();
|
||||
|
||||
struct CappedMultipartStagingGuard {
|
||||
upload_id_path: String,
|
||||
permit: Option<tokio::sync::OwnedSemaphorePermit>,
|
||||
}
|
||||
|
||||
impl Drop for CappedMultipartStagingGuard {
|
||||
fn drop(&mut self) {
|
||||
// Release the permit before checking the Arc count so a concurrent
|
||||
// Abort/Complete cleanup can remove the now-unused map entry.
|
||||
self.permit.take();
|
||||
remove_capped_multipart_staging_semaphore(&self.upload_id_path);
|
||||
}
|
||||
}
|
||||
|
||||
fn capped_multipart_staging_semaphore(upload_id_path: &str) -> Arc<tokio::sync::Semaphore> {
|
||||
CAPPED_MULTIPART_STAGING
|
||||
.get_or_init(|| Mutex::new(HashMap::new()))
|
||||
.lock()
|
||||
.expect("capped multipart staging semaphore map should not be poisoned")
|
||||
.entry(upload_id_path.to_owned())
|
||||
.or_insert_with(|| Arc::new(tokio::sync::Semaphore::new(1)))
|
||||
.clone()
|
||||
}
|
||||
|
||||
fn remove_capped_multipart_staging_semaphore(upload_id_path: &str) {
|
||||
if let Some(map) = CAPPED_MULTIPART_STAGING.get() {
|
||||
let mut map = map
|
||||
.lock()
|
||||
.expect("capped multipart staging semaphore map should not be poisoned");
|
||||
let removable = map
|
||||
.get(upload_id_path)
|
||||
.is_some_and(|semaphore| Arc::strong_count(semaphore) == 1);
|
||||
if removable {
|
||||
map.remove(upload_id_path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn multipart_size_limit_from_metadata(metadata: &HashMap<String, String>) -> Result<Option<u64>> {
|
||||
if !contains_key_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE) {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
let Some(value) = get_consistent_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE) else {
|
||||
return Err(Error::InvalidArgument(
|
||||
"multipart upload".to_string(),
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
|
||||
"missing or conflicting internal size limit".to_string(),
|
||||
));
|
||||
};
|
||||
|
||||
let limit = value.parse::<u64>().map_err(|_| {
|
||||
Error::InvalidArgument(
|
||||
"multipart upload".to_string(),
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
|
||||
"invalid internal size limit".to_string(),
|
||||
)
|
||||
})?;
|
||||
Ok(Some(limit))
|
||||
}
|
||||
|
||||
fn admitted_multipart_size(current: u64, candidate: u64, limit: u64) -> Result<u64> {
|
||||
let total = current.checked_add(candidate).ok_or_else(|| {
|
||||
Error::InvalidArgument(
|
||||
"multipart upload".to_string(),
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
|
||||
"logical size overflow".to_string(),
|
||||
)
|
||||
})?;
|
||||
if total > limit {
|
||||
return Err(Error::EntityTooLarge(total, limit));
|
||||
}
|
||||
Ok(total)
|
||||
}
|
||||
|
||||
pub(crate) struct StaleMultipartCleanupGuard {
|
||||
file_info: FileInfo,
|
||||
upload_path: String,
|
||||
@@ -194,13 +115,8 @@ impl StaleMultipartCleanupGuard {
|
||||
|
||||
pub(crate) async fn delete(self, set: &SetDisks) -> Result<()> {
|
||||
fence_commit_on_lock_loss(Some(&self.lock_guard), "stale_multipart_cleanup", &self.upload_path)?;
|
||||
let result = set
|
||||
.delete_all_with_quorum(RUSTFS_META_MULTIPART_BUCKET, &self.upload_path, self.write_quorum)
|
||||
.await;
|
||||
if result.is_ok() {
|
||||
remove_capped_multipart_staging_semaphore(&self.upload_path);
|
||||
}
|
||||
result
|
||||
set.delete_all_with_quorum(RUSTFS_META_MULTIPART_BUCKET, &self.upload_path, self.write_quorum)
|
||||
.await
|
||||
}
|
||||
}
|
||||
|
||||
@@ -449,7 +365,7 @@ fn fence_commit_on_lock_loss(guard: Option<&ObjectLockDiagGuard>, mode: &'static
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn multipart_bucket_incarnation_id(metadata: &HashMap<String, String>) -> Result<Option<Uuid>> {
|
||||
fn multipart_bucket_incarnation_id<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> Result<Option<Uuid>> {
|
||||
let Some(value) = rustfs_utils::http::metadata_compat::get_consistent_str(metadata, SUFFIX_BUCKET_INCARNATION_ID) else {
|
||||
if rustfs_utils::http::metadata_compat::contains_key_str(metadata, SUFFIX_BUCKET_INCARNATION_ID) {
|
||||
return Err(Error::other("invalid multipart bucket incarnation metadata"));
|
||||
@@ -463,12 +379,12 @@ fn multipart_bucket_incarnation_id(metadata: &HashMap<String, String>) -> Result
|
||||
Ok(Some(incarnation))
|
||||
}
|
||||
|
||||
fn multipart_bucket_incarnation_matches(metadata: &HashMap<String, String>, expected: Uuid) -> bool {
|
||||
fn multipart_bucket_incarnation_matches<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>, expected: Uuid) -> bool {
|
||||
matches!(multipart_bucket_incarnation_id(metadata), Ok(Some(actual)) if actual == expected)
|
||||
}
|
||||
|
||||
fn validate_multipart_bucket_incarnation(
|
||||
metadata: &HashMap<String, String>,
|
||||
fn validate_multipart_bucket_incarnation<S: std::hash::BuildHasher>(
|
||||
metadata: &HashMap<String, String, S>,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
upload_id: &str,
|
||||
@@ -719,61 +635,6 @@ async fn multipart_upload_paths_on_disk(disk: DiskStore, bucket: &str) -> disk::
|
||||
}
|
||||
|
||||
impl SetDisks {
|
||||
async fn current_multipart_logical_size(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
upload_id: &str,
|
||||
upload_id_path: &str,
|
||||
fi: &FileInfo,
|
||||
replacing_part: usize,
|
||||
) -> Result<u64> {
|
||||
let online_disks = self.get_disks_internal().await;
|
||||
let read_quorum = fi.read_quorum(self.default_read_quorum());
|
||||
let part_path = format!(
|
||||
"{}{}",
|
||||
path_join_buf(&[
|
||||
upload_id_path,
|
||||
fi.data_dir.map(|v| v.to_string()).unwrap_or_default().as_str(),
|
||||
]),
|
||||
SLASH_SEPARATOR
|
||||
);
|
||||
let part_numbers = match Self::list_parts(&online_disks, &part_path, read_quorum).await {
|
||||
Ok(parts) => parts,
|
||||
Err(DiskError::FileNotFound) => return Ok(0),
|
||||
Err(err) => return Err(to_object_err(err.into(), vec![bucket, object, upload_id])),
|
||||
};
|
||||
if part_numbers.is_empty() {
|
||||
return Ok(0);
|
||||
}
|
||||
let part_meta_paths = part_numbers
|
||||
.iter()
|
||||
.map(|number| format!("{part_path}part.{number}.meta"))
|
||||
.collect::<Vec<_>>();
|
||||
let existing_parts =
|
||||
Self::read_parts(&online_disks, RUSTFS_META_MULTIPART_BUCKET, &part_meta_paths, &part_numbers, read_quorum)
|
||||
.await
|
||||
.map_err(|err| to_object_err(err.into(), vec![bucket, object, upload_id]))?;
|
||||
|
||||
existing_parts.into_iter().try_fold(0_u64, |total, part| {
|
||||
if part.error.is_some() || part.number == replacing_part {
|
||||
return if part.error.is_some() {
|
||||
Err(Error::PartMissingOrCorrupt)
|
||||
} else {
|
||||
Ok(total)
|
||||
};
|
||||
}
|
||||
let part_size = u64::try_from(part.actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
|
||||
total.checked_add(part_size).ok_or_else(|| {
|
||||
Error::InvalidArgument(
|
||||
"multipart upload".to_string(),
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
|
||||
"logical size overflow".to_string(),
|
||||
)
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
async fn discover_multipart_upload_paths(
|
||||
&self,
|
||||
orig_bucket: &str,
|
||||
@@ -1269,18 +1130,9 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
crate::hp_guard!("SetDisks::put_object_part");
|
||||
let upload_id_path = Self::get_multipart_upload_dir(bucket, object, upload_id, opts.data_movement);
|
||||
|
||||
let (fi, _) = match self
|
||||
let (fi, _) = self
|
||||
.check_upload_id_exists_with_opts(bucket, object, upload_id, true, opts)
|
||||
.await
|
||||
{
|
||||
Ok(value) => value,
|
||||
Err(err @ Error::InvalidUploadID(..)) => {
|
||||
remove_capped_multipart_staging_semaphore(&upload_id_path);
|
||||
return Err(err);
|
||||
}
|
||||
Err(err) => return Err(err),
|
||||
};
|
||||
let multipart_size_limit = multipart_size_limit_from_metadata(&fi.metadata)?;
|
||||
.await?;
|
||||
ensure_data_movement_upload_access(&fi, bucket, object, upload_id, opts)?;
|
||||
ensure_multipart_bucket_incarnation(&self.ctx, &fi, bucket, object, upload_id, opts.expected_bucket_incarnation_id)
|
||||
.await?;
|
||||
@@ -1313,44 +1165,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
let part_suffix = format!("part.{part_id}");
|
||||
let tmp_part = format!("{}x{}", Uuid::new_v4(), OffsetDateTime::now_utc().unix_timestamp());
|
||||
let tmp_part_path = Arc::new(format!("{tmp_part}/{part_suffix}"));
|
||||
let part_path = format!("{}/{}/{}", upload_id_path, fi.data_dir.unwrap_or_default(), part_suffix);
|
||||
let part_lock_path = format!("{upload_id_path}/{part_suffix}");
|
||||
|
||||
// Keep at most one capped part staging locally per upload. The
|
||||
// distributed lock below is held only for the durable admission check;
|
||||
// it is reacquired for the short final rename, so Complete/Abort are
|
||||
// not blocked behind a slow body upload.
|
||||
let _capped_staging_guard = if multipart_size_limit.is_some() {
|
||||
Some(CappedMultipartStagingGuard {
|
||||
upload_id_path: upload_id_path.clone(),
|
||||
permit: Some(
|
||||
capped_multipart_staging_semaphore(&upload_id_path)
|
||||
.acquire_owned()
|
||||
.await
|
||||
.map_err(|_| Error::other("capped multipart staging semaphore closed"))?,
|
||||
),
|
||||
})
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
if let Some(limit) = multipart_size_limit {
|
||||
let admission_guard = self
|
||||
.acquire_write_lock_diag("put_object_part_admission", RUSTFS_META_MULTIPART_BUCKET, &upload_id_path)
|
||||
.await?;
|
||||
let declared_size = if data.size() >= 0 {
|
||||
u64::try_from(data.size()).map_err(|_| Error::PartMissingOrCorrupt)?
|
||||
} else if data.actual_size() >= 0 {
|
||||
u64::try_from(data.actual_size()).map_err(|_| Error::PartMissingOrCorrupt)?
|
||||
} else {
|
||||
return Err(Error::PartMissingOrCorrupt);
|
||||
};
|
||||
let current_size = self
|
||||
.current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &fi, part_id)
|
||||
.await?;
|
||||
admitted_multipart_size(current_size, declared_size, limit)?;
|
||||
drop(admission_guard);
|
||||
}
|
||||
|
||||
let result: Result<PartInfo> = async {
|
||||
let erasure =
|
||||
@@ -1521,7 +1335,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
mod_time: Some(OffsetDateTime::now_utc()),
|
||||
actual_size,
|
||||
index: index_op,
|
||||
checksums: if checksums.is_empty() { None } else { Some(checksums) },
|
||||
checksums: if checksums.is_empty() { None } else { Some(checksums.iter().map(|(k, v)| (k.clone(), v.clone())).collect()) },
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
@@ -1551,21 +1365,30 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
.await?;
|
||||
}
|
||||
|
||||
let part_path = format!("{}/{}/{}", upload_id_path, fi.data_dir.unwrap_or_default(), part_suffix);
|
||||
let part_lock_path = format!("{upload_id_path}/{part_suffix}");
|
||||
|
||||
#[cfg(test)]
|
||||
pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockAcquire).await;
|
||||
// Capped uploads reacquire the upload-wide write lock for the
|
||||
// final durable check and rename. Uncapped uploads retain the
|
||||
// concurrent encode path and only serialize the final same-part
|
||||
// rename; completion/abort use the upload-wide write lock.
|
||||
let (_upload_commit_guard, _part_commit_guard) = if multipart_size_limit.is_some() {
|
||||
let upload_guard = self
|
||||
.acquire_write_lock_diag("put_object_part_commit", RUSTFS_META_MULTIPART_BUCKET, &upload_id_path)
|
||||
.await?;
|
||||
let part_guard = self
|
||||
.acquire_write_lock_diag("put_object_part_commit", RUSTFS_META_MULTIPART_BUCKET, &part_lock_path)
|
||||
.await?;
|
||||
(Some(upload_guard), Some(part_guard))
|
||||
} else if opts.no_lock {
|
||||
// Serialize only same-part commits (rename_part), not the whole upload.
|
||||
// Each concurrent stream writes to its own unique temp dir (see
|
||||
// `tmp_part` above), so the encode/stream phase never conflicts and must
|
||||
// stay lock-free — holding a lock across it would serialize slow
|
||||
// re-transmits of the same part and defeat the S3 "last finisher wins"
|
||||
// semantics. The mixed-generation hazard is confined to rename_part,
|
||||
// where two temp parts are moved cross-disk onto the SAME final
|
||||
// part_path: interleaving there can leave shards from two generations,
|
||||
// each individually bitrot-valid, that only surface as silent corruption
|
||||
// at read time (backlog#853). A write lock scoped to this part number
|
||||
// makes each same-part commit atomic across disks, so the last committer
|
||||
// wins consistently, while different part numbers commit onto disjoint
|
||||
// part paths and stay concurrent (issue#5961 — an uploadId-wide write
|
||||
// lock serialized them into 503 lock-acquire timeouts). The shared
|
||||
// uploadId read lock keeps completion/abort (which take the uploadId
|
||||
// write lock) from racing any in-flight part commit; a guarded
|
||||
// completion takes the object lock before the upload lock to preserve
|
||||
// global ordering.
|
||||
let (_upload_commit_guard, _part_commit_guard) = if opts.no_lock {
|
||||
(None, None)
|
||||
} else {
|
||||
let upload_guard = self
|
||||
@@ -1577,16 +1400,8 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
(Some(upload_guard), Some(part_guard))
|
||||
};
|
||||
let (commit_fi, _) = self
|
||||
.check_upload_id_exists_with_opts(bucket, object, upload_id, multipart_size_limit.is_some(), opts)
|
||||
.check_upload_id_exists_with_opts(bucket, object, upload_id, false, opts)
|
||||
.await?;
|
||||
let commit_size_limit = multipart_size_limit_from_metadata(&commit_fi.metadata)?;
|
||||
if commit_size_limit != multipart_size_limit {
|
||||
return Err(Error::InvalidArgument(
|
||||
"multipart upload".to_string(),
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
|
||||
"size limit metadata changed or is missing".to_string(),
|
||||
));
|
||||
}
|
||||
ensure_data_movement_upload_access(&commit_fi, bucket, object, upload_id, opts)?;
|
||||
ensure_multipart_bucket_incarnation(
|
||||
&self.ctx,
|
||||
@@ -1616,14 +1431,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
}
|
||||
ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
|
||||
|
||||
if let Some(limit) = commit_size_limit {
|
||||
let current_size = self
|
||||
.current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &commit_fi, part_id)
|
||||
.await?;
|
||||
let candidate_size = u64::try_from(actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
|
||||
admitted_multipart_size(current_size, candidate_size, limit)?;
|
||||
}
|
||||
|
||||
let _ = self
|
||||
.rename_part(
|
||||
&shuffle_disks,
|
||||
@@ -1716,7 +1523,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
max_parts,
|
||||
part_number_marker,
|
||||
user_defined: {
|
||||
let mut metadata = fi.metadata.clone();
|
||||
let mut metadata: HashMap<String, String> = fi.metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
|
||||
strip_internal_multipart_metadata(&mut metadata);
|
||||
metadata
|
||||
},
|
||||
@@ -1975,7 +1782,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
let mod_time = opts.mod_time.unwrap_or_else(OffsetDateTime::now_utc);
|
||||
|
||||
for f in parts_metadatas.iter_mut() {
|
||||
f.metadata = user_defined.clone();
|
||||
f.metadata = user_defined.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
|
||||
f.mod_time = Some(mod_time);
|
||||
f.fresh = true;
|
||||
}
|
||||
@@ -2064,7 +1871,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
upload_id: upload_id.to_owned(),
|
||||
user_defined: {
|
||||
strip_internal_multipart_metadata(&mut fi.metadata);
|
||||
fi.metadata.clone()
|
||||
fi.metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect()
|
||||
},
|
||||
..Default::default()
|
||||
})
|
||||
@@ -2084,17 +1891,12 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
|
||||
let upload_id_path = Self::get_multipart_upload_dir(bucket, object, upload_id, opts.data_movement);
|
||||
|
||||
let result = self
|
||||
.delete_all_with_quorum(
|
||||
RUSTFS_META_MULTIPART_BUCKET,
|
||||
&upload_id_path,
|
||||
fi.write_quorum(self.default_write_quorum()),
|
||||
)
|
||||
.await;
|
||||
if result.is_ok() {
|
||||
remove_capped_multipart_staging_semaphore(&upload_id_path);
|
||||
}
|
||||
result
|
||||
self.delete_all_with_quorum(
|
||||
RUSTFS_META_MULTIPART_BUCKET,
|
||||
&upload_id_path,
|
||||
fi.write_quorum(self.default_write_quorum()),
|
||||
)
|
||||
.await
|
||||
}
|
||||
// complete_multipart_upload finished
|
||||
#[tracing::instrument(skip(self))]
|
||||
@@ -2214,27 +2016,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
return Err(Error::other("part result number err"));
|
||||
}
|
||||
|
||||
if let Some(limit) = multipart_size_limit_from_metadata(&fi.metadata)? {
|
||||
let mut total = 0_u64;
|
||||
for part in &object_parts {
|
||||
if part.error.is_some() {
|
||||
return Err(Error::PartMissingOrCorrupt);
|
||||
}
|
||||
let part_size = u64::try_from(part.actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
|
||||
total = total.checked_add(part_size).ok_or_else(|| {
|
||||
Error::InvalidArgument(
|
||||
"multipart upload".to_string(),
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
|
||||
"logical size overflow".to_string(),
|
||||
)
|
||||
})?;
|
||||
}
|
||||
if total > limit {
|
||||
return Err(Error::EntityTooLarge(total, limit));
|
||||
}
|
||||
rustfs_utils::http::metadata_compat::remove_str(&mut fi.metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE);
|
||||
}
|
||||
|
||||
let mut checksum_type = rustfs_rio::ChecksumType::NONE;
|
||||
|
||||
if let Some(cs) = fi.metadata.get(rustfs_rio::RUSTFS_MULTIPART_CHECKSUM) {
|
||||
@@ -3243,17 +3024,13 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
Ok(ObjectInfo::from_file_info(&fi, &commit_bucket, &commit_object, commit_is_versioned))
|
||||
};
|
||||
|
||||
let result = if detach_commit_owner {
|
||||
if detach_commit_owner {
|
||||
tokio::spawn(commit)
|
||||
.await
|
||||
.map_err(|err| Error::other(format!("complete_multipart_upload commit task failed: {err}")))?
|
||||
} else {
|
||||
commit.await
|
||||
};
|
||||
if result.is_ok() {
|
||||
remove_capped_multipart_staging_semaphore(&upload_id_path);
|
||||
}
|
||||
result
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3319,34 +3096,6 @@ mod tests {
|
||||
assert!(multipart_bucket_incarnation_id(&nil_metadata).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multipart_size_limit_metadata_is_dual_key_and_fail_closed() {
|
||||
let mut metadata = HashMap::new();
|
||||
insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "100".to_string());
|
||||
assert_eq!(multipart_size_limit_from_metadata(&metadata).unwrap(), Some(100));
|
||||
|
||||
metadata.insert("x-minio-internal-max-total-object-size".to_string(), "101".to_string());
|
||||
assert!(multipart_size_limit_from_metadata(&metadata).is_err());
|
||||
|
||||
let mut invalid = HashMap::new();
|
||||
insert_str(&mut invalid, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "-1".to_string());
|
||||
assert!(multipart_size_limit_from_metadata(&invalid).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multipart_size_admission_handles_boundaries_and_overflow() {
|
||||
assert_eq!(admitted_multipart_size(90, 10, 100).unwrap(), 100);
|
||||
assert!(matches!(
|
||||
admitted_multipart_size(90, 11, 100),
|
||||
Err(StorageError::EntityTooLarge(101, 100))
|
||||
));
|
||||
assert!(admitted_multipart_size(u64::MAX - 1, 1, u64::MAX).is_ok());
|
||||
assert!(matches!(
|
||||
admitted_multipart_size(u64::MAX, 1, u64::MAX),
|
||||
Err(StorageError::InvalidArgument(_, _, _))
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multipart_bucket_incarnation_gate_accepts_only_current_or_same_lifetime_legacy_uploads() {
|
||||
let expected = Uuid::new_v4();
|
||||
|
||||
@@ -19,6 +19,7 @@
|
||||
//! bounds are unchanged, and the impls reach shared primitives through the
|
||||
//! SetDisks core (io_primitives) via inherent calls.
|
||||
|
||||
use ahash::AHashMap;
|
||||
#[cfg(test)]
|
||||
use super::super::MetadataCacheInvalidationProbe;
|
||||
use super::super::{
|
||||
@@ -1107,13 +1108,13 @@ fn is_restore_control_metadata(key: &str) -> bool {
|
||||
.is_some_and(|remainder| remainder.is_empty())
|
||||
}
|
||||
|
||||
fn restore_metadata_update_preserves_protected_metadata(
|
||||
existing: &HashMap<String, String>,
|
||||
replacement: &HashMap<String, String>,
|
||||
fn restore_metadata_update_preserves_protected_metadata<S1: std::hash::BuildHasher, S2: std::hash::BuildHasher>(
|
||||
existing: &HashMap<String, String, S1>,
|
||||
replacement: &HashMap<String, String, S2>,
|
||||
) -> bool {
|
||||
let mut existing = existing.clone();
|
||||
let mut existing: HashMap<String, String> = existing.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
|
||||
clean_metadata(&mut existing);
|
||||
let mut replacement = replacement.clone();
|
||||
let mut replacement: HashMap<String, String> = replacement.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
|
||||
clean_metadata(&mut replacement);
|
||||
let existing_count = existing.keys().filter(|key| !is_restore_control_metadata(key)).count();
|
||||
let replacement_count = replacement.keys().filter(|key| !is_restore_control_metadata(key)).count();
|
||||
@@ -2211,9 +2212,9 @@ pub(in crate::set_disk) fn stored_replication_category_metadata(existing: &Objec
|
||||
///
|
||||
/// Returns whether `inbound` was modified. Callers must hold the object write
|
||||
/// lock so the stored values compared here are the ones being replaced.
|
||||
pub(in crate::set_disk) fn merge_replication_metadata_lww(
|
||||
inbound: &mut HashMap<String, String>,
|
||||
existing: &HashMap<String, String>,
|
||||
pub(in crate::set_disk) fn merge_replication_metadata_lww<S1: std::hash::BuildHasher, S2: std::hash::BuildHasher>(
|
||||
inbound: &mut HashMap<String, String, S1>,
|
||||
existing: &HashMap<String, String, S2>,
|
||||
opts: &ObjectOptions,
|
||||
) -> bool {
|
||||
use rustfs_utils::http::headers::{
|
||||
@@ -2844,7 +2845,7 @@ impl SetDisks {
|
||||
)));
|
||||
}
|
||||
|
||||
fi.metadata = user_defined;
|
||||
fi.metadata = user_defined.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
|
||||
fi.mod_time = mod_time;
|
||||
fi.size = w_size as i64;
|
||||
fi.versioned = opts.versioned || opts.version_suspended;
|
||||
@@ -6051,7 +6052,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let mut replacement_metadata = (*src_info.user_defined).clone();
|
||||
let mut replacement_metadata: AHashMap<String, String> = (*src_info.user_defined).iter().map(|(k, v)| (k.clone(), v.clone())).collect();
|
||||
if let Some(part_checksums) = preserved_part_checksums {
|
||||
rustfs_utils::http::insert_str(&mut replacement_metadata, rustfs_utils::http::SUFFIX_PART_CHECKSUMS, part_checksums);
|
||||
}
|
||||
@@ -7493,7 +7494,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str(),
|
||||
X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str(),
|
||||
] {
|
||||
if let Some(value) = fi.metadata.lookup(header).filter(|value| !value.is_empty()) {
|
||||
if let Some(value) = fi.metadata.get(header).filter(|value| !value.is_empty()) {
|
||||
transition_meta.insert(header.to_ascii_lowercase(), value.to_string());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -99,8 +99,6 @@ fn preflight_startup_rpc_secret_with(
|
||||
const LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES: usize = 6;
|
||||
const LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY: Duration = Duration::from_secs(60 * 3);
|
||||
const LOCAL_DECOMMISSION_RESUME_RETRY_DELAY: Duration = Duration::from_secs(30);
|
||||
const REBALANCE_INITIAL_RESUME_DELAY: Duration = Duration::from_secs(10);
|
||||
const REBALANCE_RESUME_RETRY_DELAY: Duration = Duration::from_secs(10);
|
||||
|
||||
fn should_retry_local_decommission_resume(err: &Error, attempt: usize) -> bool {
|
||||
matches!(err, Error::ConfigNotFound) && attempt < LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES
|
||||
@@ -110,12 +108,8 @@ fn should_retry_format_load(err: &Error) -> bool {
|
||||
!matches!(err, Error::CorruptedFormat)
|
||||
}
|
||||
|
||||
fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_resume_required: bool) -> bool {
|
||||
rebalance_resume_required && !decommission_running
|
||||
}
|
||||
|
||||
fn should_defer_rebalance_auto_start(distributed: bool, fleet_proof_available: bool) -> bool {
|
||||
distributed && !fleet_proof_available
|
||||
fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_meta_loaded: bool) -> bool {
|
||||
rebalance_meta_loaded && !decommission_running
|
||||
}
|
||||
|
||||
fn should_schedule_local_decommission_resume(
|
||||
@@ -133,17 +127,6 @@ async fn wait_for_local_decommission_resume_delay(rx: &CancellationToken, delay:
|
||||
}
|
||||
}
|
||||
|
||||
async fn wait_for_rebalance_resume_delay(rx: &CancellationToken, delay: Duration) -> bool {
|
||||
tokio::select! {
|
||||
_ = rx.cancelled() => false,
|
||||
_ = tokio::time::sleep(delay) => true,
|
||||
}
|
||||
}
|
||||
|
||||
async fn wait_for_rebalance_resume_retry(rx: &CancellationToken) -> bool {
|
||||
wait_for_rebalance_resume_delay(rx, REBALANCE_RESUME_RETRY_DELAY).await
|
||||
}
|
||||
|
||||
fn resolve_store_init_stage_result(result: Result<()>, stage: &str) -> Result<()> {
|
||||
result.map_err(|err| Error::other(format!("store init failed during {stage}: {err}")))
|
||||
}
|
||||
@@ -300,71 +283,6 @@ async fn resume_local_decommission_after_init(store: Arc<ECStore>, rx: Cancellat
|
||||
}
|
||||
}
|
||||
|
||||
async fn resume_rebalance_after_init(store: Arc<ECStore>, rx: CancellationToken) {
|
||||
if !wait_for_rebalance_resume_delay(&rx, REBALANCE_INITIAL_RESUME_DELAY).await {
|
||||
return;
|
||||
}
|
||||
|
||||
loop {
|
||||
if rx.is_cancelled() {
|
||||
return;
|
||||
}
|
||||
|
||||
let resume_required = store
|
||||
.rebalance_meta
|
||||
.read()
|
||||
.await
|
||||
.as_ref()
|
||||
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation);
|
||||
if !resume_required {
|
||||
return;
|
||||
}
|
||||
|
||||
if should_defer_rebalance_auto_start(
|
||||
store.ctx.is_dist_erasure().await,
|
||||
crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_some(),
|
||||
) {
|
||||
if !wait_for_rebalance_resume_retry(&rx).await {
|
||||
return;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
match store.start_rebalance().await {
|
||||
Ok(()) => return,
|
||||
Err(err) if crate::core::pools::is_pool_activation_fleet_proof_error(&err) => {
|
||||
warn!(
|
||||
event = EVENT_ECSTORE_INIT_STATUS,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_STORE_INIT,
|
||||
stage = "start_rebalance",
|
||||
state = "retrying",
|
||||
reason = "fleet_capability_proof_unavailable",
|
||||
error = %err,
|
||||
retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
|
||||
"Retrying deferred rebalance auto-start"
|
||||
);
|
||||
if !wait_for_rebalance_resume_retry(&rx).await {
|
||||
return;
|
||||
}
|
||||
}
|
||||
Err(err) => {
|
||||
error!(
|
||||
event = EVENT_ECSTORE_INIT_STATUS,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_STORE_INIT,
|
||||
stage = "start_rebalance",
|
||||
state = "failed",
|
||||
reason = "deferred_resume_failed",
|
||||
error = %err,
|
||||
"Failed to resume rebalance after store initialization"
|
||||
);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl ECStore {
|
||||
/// Validate topology and process storage-class overrides before any disk is opened.
|
||||
pub fn validate_startup_storage_class(endpoint_pools: &EndpointServerPools) -> Result<()> {
|
||||
@@ -656,49 +574,12 @@ impl ECStore {
|
||||
}
|
||||
|
||||
resolve_store_init_stage_result(self.load_rebalance_meta().await, "load_rebalance_meta")?;
|
||||
let rebalance_resume_required = {
|
||||
let rebalance_meta = self.rebalance_meta.read().await;
|
||||
rebalance_meta
|
||||
.as_ref()
|
||||
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
|
||||
};
|
||||
let rebalance_meta_loaded = self.rebalance_meta.read().await.is_some();
|
||||
let decommission_running =
|
||||
pool_meta_has_active_decommission(&installed_pool_meta) || self.is_decommission_running().await;
|
||||
let distributed = self.ctx.is_dist_erasure().await;
|
||||
let fleet_proof_available = crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_some();
|
||||
let mut rebalance_auto_start_deferred = false;
|
||||
if should_auto_start_rebalance_after_init(decommission_running, rebalance_resume_required) {
|
||||
if should_defer_rebalance_auto_start(distributed, fleet_proof_available) {
|
||||
rebalance_auto_start_deferred = true;
|
||||
warn!(
|
||||
event = EVENT_ECSTORE_INIT_STATUS,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_STORE_INIT,
|
||||
stage = "start_rebalance",
|
||||
state = "deferred",
|
||||
reason = "fleet_capability_proof_unavailable",
|
||||
retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
|
||||
"Deferred rebalance auto-start until a live fleet capability proof is available"
|
||||
);
|
||||
} else if let Err(err) = self.start_rebalance().await {
|
||||
if crate::core::pools::is_pool_activation_fleet_proof_error(&err) {
|
||||
rebalance_auto_start_deferred = true;
|
||||
warn!(
|
||||
event = EVENT_ECSTORE_INIT_STATUS,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_STORE_INIT,
|
||||
stage = "start_rebalance",
|
||||
state = "deferred",
|
||||
reason = "fleet_capability_proof_changed",
|
||||
error = %err,
|
||||
retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
|
||||
"Deferred rebalance auto-start after the fleet capability proof changed"
|
||||
);
|
||||
} else {
|
||||
return resolve_store_init_stage_result(Err(err), "start_rebalance");
|
||||
}
|
||||
}
|
||||
} else if decommission_running && rebalance_resume_required {
|
||||
if should_auto_start_rebalance_after_init(decommission_running, rebalance_meta_loaded) {
|
||||
resolve_store_init_stage_result(self.start_rebalance().await, "start_rebalance")?;
|
||||
} else if decommission_running && rebalance_meta_loaded {
|
||||
warn!(
|
||||
event = EVENT_ECSTORE_INIT_STATUS,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
@@ -735,13 +616,12 @@ impl ECStore {
|
||||
.is_ok();
|
||||
if should_schedule_local_decommission_resume(&local_pool_indices, pool_meta_replica_state, pool_meta_write_safe) {
|
||||
let store = self.clone();
|
||||
let decommission_rx = rx.clone();
|
||||
|
||||
tokio::spawn(async move {
|
||||
if !wait_for_local_decommission_resume_delay(&decommission_rx, LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY).await {
|
||||
if !wait_for_local_decommission_resume_delay(&rx, LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY).await {
|
||||
return;
|
||||
}
|
||||
resume_local_decommission_after_init(store, decommission_rx, local_pool_indices).await;
|
||||
resume_local_decommission_after_init(store, rx, local_pool_indices).await;
|
||||
});
|
||||
} else if !local_pool_indices.is_empty() {
|
||||
error!(
|
||||
@@ -768,11 +648,6 @@ impl ECStore {
|
||||
info!("TierConfigMgr init error: {}", err);
|
||||
}
|
||||
|
||||
if rebalance_auto_start_deferred {
|
||||
let store = self.clone();
|
||||
tokio::spawn(resume_rebalance_after_init(store, rx));
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -790,8 +665,7 @@ mod tests {
|
||||
load_pool_meta_for_startup, persist_pool_meta_for_startup_if_safe, pool_first_endpoint_is_local,
|
||||
pool_meta_has_active_decommission, preflight_startup_rpc_secret_with, resolve_startup_pool_defaults_with,
|
||||
resolve_store_init_stage_result, save_validated_pool_meta_for_startup, should_auto_start_rebalance_after_init,
|
||||
should_defer_rebalance_auto_start, should_retry_format_load, should_retry_local_decommission_resume,
|
||||
wait_for_local_decommission_resume_delay,
|
||||
should_retry_format_load, should_retry_local_decommission_resume, wait_for_local_decommission_resume_delay,
|
||||
};
|
||||
#[cfg(feature = "test-util")]
|
||||
use crate::disk::DiskAPI;
|
||||
@@ -1576,7 +1450,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_should_auto_start_rebalance_after_init_allows_active_rebalance_without_decommission() {
|
||||
fn test_should_auto_start_rebalance_after_init_allows_loaded_rebalance_without_decommission() {
|
||||
assert!(should_auto_start_rebalance_after_init(false, true));
|
||||
}
|
||||
|
||||
@@ -1586,17 +1460,10 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_should_auto_start_rebalance_after_init_rejects_terminal_or_missing_rebalance() {
|
||||
fn test_should_auto_start_rebalance_after_init_rejects_missing_rebalance_meta() {
|
||||
assert!(!should_auto_start_rebalance_after_init(false, false));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_should_defer_rebalance_auto_start_only_without_distributed_fleet_proof() {
|
||||
assert!(should_defer_rebalance_auto_start(true, false));
|
||||
assert!(!should_defer_rebalance_auto_start(true, true));
|
||||
assert!(!should_defer_rebalance_auto_start(false, false));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_store_init_recovery_skips_rebalance_when_decommission_metadata_is_active() {
|
||||
let pool_meta = init_test_pool_meta(Some(PoolDecommissionInfo {
|
||||
@@ -1606,69 +1473,22 @@ mod tests {
|
||||
canceled: false,
|
||||
..Default::default()
|
||||
}));
|
||||
let rebalance_meta = Some(RebalanceMeta {
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Started,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
});
|
||||
let rebalance_meta = Some(RebalanceMeta::default());
|
||||
|
||||
assert!(!should_auto_start_rebalance_after_init(
|
||||
pool_meta_has_active_decommission(&pool_meta),
|
||||
rebalance_meta
|
||||
.as_ref()
|
||||
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
|
||||
rebalance_meta.is_some()
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_store_init_recovery_allows_active_rebalance_without_decommission() {
|
||||
fn test_store_init_recovery_allows_rebalance_when_only_rebalance_metadata_exists() {
|
||||
let pool_meta = init_test_pool_meta(None);
|
||||
let rebalance_meta = Some(RebalanceMeta {
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Started,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
});
|
||||
let rebalance_meta = Some(RebalanceMeta::default());
|
||||
|
||||
assert!(should_auto_start_rebalance_after_init(
|
||||
pool_meta_has_active_decommission(&pool_meta),
|
||||
rebalance_meta
|
||||
.as_ref()
|
||||
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_store_init_recovery_skips_completed_rebalance_metadata() {
|
||||
let pool_meta = init_test_pool_meta(None);
|
||||
let rebalance_meta = Some(RebalanceMeta {
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Completed,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
});
|
||||
|
||||
assert!(!should_auto_start_rebalance_after_init(
|
||||
pool_meta_has_active_decommission(&pool_meta),
|
||||
rebalance_meta
|
||||
.as_ref()
|
||||
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
|
||||
rebalance_meta.is_some()
|
||||
));
|
||||
}
|
||||
|
||||
|
||||
@@ -26,18 +26,18 @@ static STRICT_BUCKET_NAME_REGEX: LazyLock<Regex> =
|
||||
static NON_STRICT_BUCKET_NAME_REGEX: LazyLock<Regex> =
|
||||
LazyLock::new(|| Regex::new(r"^[A-Za-z0-9][A-Za-z0-9\.\-_:]{1,61}[A-Za-z0-9]$").expect("valid non-strict bucket name regex"));
|
||||
|
||||
pub fn clean_metadata(metadata: &mut HashMap<String, String>) {
|
||||
pub fn clean_metadata<S: std::hash::BuildHasher>(metadata: &mut HashMap<String, String, S>) {
|
||||
remove_standard_storage_class(metadata);
|
||||
clean_metadata_keys(metadata, &["md5Sum", "etag", "expires", AMZ_OBJECT_TAGGING, "last-modified"]);
|
||||
}
|
||||
|
||||
pub fn remove_standard_storage_class(metadata: &mut HashMap<String, String>) {
|
||||
pub fn remove_standard_storage_class<S: std::hash::BuildHasher>(metadata: &mut HashMap<String, String, S>) {
|
||||
if metadata.get(AMZ_STORAGE_CLASS) == Some(&STANDARD.to_string()) {
|
||||
metadata.remove(AMZ_STORAGE_CLASS);
|
||||
}
|
||||
}
|
||||
|
||||
pub fn clean_metadata_keys(metadata: &mut HashMap<String, String>, key_names: &[&str]) {
|
||||
pub fn clean_metadata_keys<S: std::hash::BuildHasher>(metadata: &mut HashMap<String, String, S>, key_names: &[&str]) {
|
||||
for key in key_names {
|
||||
metadata.remove(key.to_owned());
|
||||
}
|
||||
|
||||
@@ -51,6 +51,9 @@ s3s = { workspace = true, features = ["minio"] }
|
||||
regex.workspace = true
|
||||
arc-swap.workspace = true
|
||||
|
||||
# High-performance hashing
|
||||
ahash = { workspace = true, features = ["serde"] }
|
||||
|
||||
[dev-dependencies]
|
||||
criterion = { workspace = true, features = ["html_reports"] }
|
||||
tempfile = { workspace = true }
|
||||
|
||||
@@ -22,6 +22,7 @@ use rustfs_utils::http::{
|
||||
contains_key_str, get_consistent_str, get_str, has_internal_suffix, insert_str, is_encryption_metadata_key,
|
||||
starts_with_ignore_ascii_case,
|
||||
};
|
||||
use ahash::AHashMap;
|
||||
use s3s::dto::{RestoreStatus, Timestamp};
|
||||
use s3s::header::X_AMZ_RESTORE;
|
||||
use serde::de::{self, MapAccess, SeqAccess, Visitor, value::MapAccessDeserializer};
|
||||
@@ -67,7 +68,7 @@ pub struct ObjectPartInfo {
|
||||
// Index holds the index of the part in the erasure coding
|
||||
pub index: Option<Bytes>,
|
||||
// Checksums holds checksums of the part
|
||||
pub checksums: Option<HashMap<String, String>>,
|
||||
pub checksums: Option<AHashMap<String, String>>,
|
||||
pub error: Option<String>,
|
||||
}
|
||||
|
||||
@@ -268,7 +269,7 @@ pub struct FileInfo {
|
||||
pub mode: Option<u32>,
|
||||
// WrittenByVersion is the unix time stamp of the version that created this version of the object
|
||||
pub written_by_version: Option<u64>,
|
||||
pub metadata: HashMap<String, String>,
|
||||
pub metadata: AHashMap<String, String>,
|
||||
pub parts: Vec<ObjectPartInfo>,
|
||||
pub erasure: ErasureInfo,
|
||||
// MarkDeleted marks this version as deleted
|
||||
@@ -301,7 +302,7 @@ fn is_sensitive_metadata_key(key: &str) -> bool {
|
||||
.any(|prefix| starts_with_ignore_ascii_case(key, prefix))
|
||||
}
|
||||
|
||||
struct RedactedMetadata<'a>(&'a HashMap<String, String>);
|
||||
struct RedactedMetadata<'a>(&'a AHashMap<String, String>);
|
||||
|
||||
impl std::fmt::Debug for RedactedMetadata<'_> {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
@@ -425,7 +426,7 @@ struct FileInfoMapDef {
|
||||
size: i64,
|
||||
mode: Option<u32>,
|
||||
written_by_version: Option<u64>,
|
||||
metadata: HashMap<String, String>,
|
||||
metadata: AHashMap<String, String>,
|
||||
parts: Vec<ObjectPartInfo>,
|
||||
erasure: ErasureInfo,
|
||||
mark_deleted: bool,
|
||||
@@ -1079,7 +1080,7 @@ impl FileInfo {
|
||||
mod_time: Option<OffsetDateTime>,
|
||||
actual_size: i64,
|
||||
index: Option<Bytes>,
|
||||
checksums: Option<HashMap<String, String>>,
|
||||
checksums: Option<AHashMap<String, String>>,
|
||||
) {
|
||||
let part = ObjectPartInfo {
|
||||
etag,
|
||||
@@ -1457,7 +1458,7 @@ pub fn parse_restore_obj_status(restore_hdr: &str) -> Result<RestoreStatus> {
|
||||
Err(Error::other(ERR_RESTORE_HDR_MALFORMED))
|
||||
}
|
||||
|
||||
pub fn is_restored_object_on_disk(meta: &HashMap<String, String>) -> bool {
|
||||
pub fn is_restored_object_on_disk<S: std::hash::BuildHasher>(meta: &HashMap<String, String, S>) -> bool {
|
||||
if let Some(restore_hdr) = meta.get(X_AMZ_RESTORE.as_str())
|
||||
&& let Ok(restore_status) = parse_restore_obj_status(restore_hdr)
|
||||
{
|
||||
@@ -2135,7 +2136,7 @@ mod tests {
|
||||
-1_000_000i64..=1_000_000i64,
|
||||
optional_timestamp_strategy(),
|
||||
proptest::option::of(bytes_strategy(16)),
|
||||
proptest::option::of(hash_map(small_string_strategy(), small_string_strategy(), 0..=3)),
|
||||
proptest::option::of(hash_map(small_string_strategy(), small_string_strategy(), 0..=3).prop_map(|m| m.into_iter().collect::<AHashMap<String, String>>())),
|
||||
proptest::option::of(small_string_strategy()),
|
||||
)
|
||||
.prop_map(|(etag, number, size, actual_size, mod_time, index, checksums, error)| ObjectPartInfo {
|
||||
@@ -2170,7 +2171,7 @@ mod tests {
|
||||
-1_000_000i64..=1_000_000i64,
|
||||
proptest::option::of(any::<u32>()),
|
||||
proptest::option::of(any::<u64>()),
|
||||
hash_map(small_string_strategy(), small_string_strategy(), 0..=4),
|
||||
hash_map(small_string_strategy(), small_string_strategy(), 0..=4).prop_map(|m| m.into_iter().collect::<AHashMap<String, String>>()),
|
||||
vec(object_part_info_strategy(), 0..=3),
|
||||
erasure_info_strategy(),
|
||||
any::<bool>(),
|
||||
|
||||
@@ -174,10 +174,10 @@ fn valid_target_delete_marker_version(arn: &str, version_id: &str) -> bool {
|
||||
/// included in the quorum hash, so such a divergence does surface — but as a
|
||||
/// quorum failure on an otherwise healthy object, which is not a state worth
|
||||
/// reaching. Merge the RPC metadata carrier instead, and only ever insert.
|
||||
fn persist_target_delete_marker_versions(
|
||||
fn persist_target_delete_marker_versions<S: std::hash::BuildHasher>(
|
||||
meta_sys: &mut HashMap<String, Vec<u8>>,
|
||||
versions: &HashMap<String, String>,
|
||||
transport_metadata: &HashMap<String, String>,
|
||||
transport_metadata: &HashMap<String, String, S>,
|
||||
) {
|
||||
let mut bounded = BTreeMap::new();
|
||||
// A corrupt carrier means the dual internal prefixes disagreed. Do not merge
|
||||
|
||||
@@ -181,7 +181,7 @@ mod tests {
|
||||
data_dir: Some(data_dir),
|
||||
size: 64 * 1024,
|
||||
mod_time: Some(OffsetDateTime::now_utc()),
|
||||
metadata,
|
||||
metadata: metadata.into_iter().collect(),
|
||||
erasure: ErasureInfo {
|
||||
algorithm: ErasureAlgo::ReedSolomon.to_string(),
|
||||
data_blocks: 4,
|
||||
|
||||
@@ -26,7 +26,7 @@ use super::msgp_decode::{
|
||||
PrependByteReader, prealloc_hint, read_exact_vec, read_nil_or_array_len, read_nil_or_map_len, skip_msgp_value,
|
||||
};
|
||||
use super::*;
|
||||
use crate::{ChecksumInfo, TransitionVersionState};
|
||||
use crate::{AHashMap, ChecksumInfo, TransitionVersionState};
|
||||
use rustfs_utils::HashAlgorithm;
|
||||
use rustfs_utils::http::{
|
||||
RUSTFS_INTERNAL_PREFIX, SUFFIX_CRC, SUFFIX_FREE_VERSION, SUFFIX_INLINE_DATA, SUFFIX_PART_CHECKSUMS, SUFFIX_PURGESTATUS,
|
||||
@@ -377,7 +377,7 @@ impl<'a> DerivedInternalMetadata<'a> {
|
||||
}
|
||||
}
|
||||
|
||||
struct UniquePartChecksums(HashMap<String, String>);
|
||||
struct UniquePartChecksums(AHashMap<String, String>);
|
||||
|
||||
impl<'de> serde::Deserialize<'de> for UniquePartChecksums {
|
||||
fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
|
||||
@@ -397,7 +397,7 @@ impl<'de> serde::Deserialize<'de> for UniquePartChecksums {
|
||||
where
|
||||
A: serde::de::SeqAccess<'de>,
|
||||
{
|
||||
let mut checksums = HashMap::with_capacity(seq.size_hint().unwrap_or_default());
|
||||
let mut checksums = AHashMap::with_capacity(seq.size_hint().unwrap_or_default());
|
||||
while let Some((key, value)) = seq.next_element::<(String, String)>()? {
|
||||
if checksums.insert(key, value).is_some() {
|
||||
return Err(serde::de::Error::custom("duplicate part checksum name"));
|
||||
@@ -1477,7 +1477,7 @@ pub struct MetaObjectV1 {
|
||||
#[serde(rename = "Erasure")]
|
||||
pub erasure: MetaObjectV1Erasure,
|
||||
#[serde(rename = "Meta")]
|
||||
pub meta: HashMap<String, String>,
|
||||
pub meta: AHashMap<String, String>,
|
||||
#[serde(rename = "Parts")]
|
||||
pub parts: Vec<MetaObjectV1Part>,
|
||||
#[serde(rename = "VersionID")]
|
||||
@@ -1543,7 +1543,7 @@ pub struct MetaObjectV1Part {
|
||||
#[serde(rename = "i")]
|
||||
pub index: Option<Bytes>,
|
||||
#[serde(rename = "crc")]
|
||||
pub checksums: Option<HashMap<String, String>>,
|
||||
pub checksums: Option<AHashMap<String, String>>,
|
||||
#[serde(rename = "err")]
|
||||
pub error: Option<String>,
|
||||
}
|
||||
@@ -1887,7 +1887,7 @@ impl MetaObjectV1Part {
|
||||
"i" => self.index = Some(Bytes::from(read_msgp_bin(rd)?)),
|
||||
"crc" => {
|
||||
let len = rmp::decode::read_map_len(rd)? as usize;
|
||||
let mut checksums = HashMap::with_capacity(prealloc_hint(len));
|
||||
let mut checksums = AHashMap::with_capacity(prealloc_hint(len));
|
||||
for _ in 0..len {
|
||||
checksums.insert(read_msgp_string(rd)?, read_msgp_string(rd)?);
|
||||
}
|
||||
@@ -2570,7 +2570,7 @@ impl MetaObject {
|
||||
Vec::new()
|
||||
};
|
||||
|
||||
let mut metadata = HashMap::with_capacity(self.meta_user.len() + self.meta_sys.len());
|
||||
let mut metadata = AHashMap::with_capacity(self.meta_user.len() + self.meta_sys.len());
|
||||
for (k, v) in &self.meta_user {
|
||||
if k == AMZ_META_UNENCRYPTED_CONTENT_LENGTH || k == AMZ_META_UNENCRYPTED_CONTENT_MD5 {
|
||||
continue;
|
||||
@@ -2861,7 +2861,7 @@ impl From<FileInfo> for MetaObject {
|
||||
}
|
||||
}
|
||||
|
||||
fn get_internal_replication_state(metadata: &HashMap<String, String>) -> Option<ReplicationState> {
|
||||
fn get_internal_replication_state<S: std::hash::BuildHasher>(metadata: &HashMap<String, String, S>) -> Option<ReplicationState> {
|
||||
let mut rs = ReplicationState::default();
|
||||
let mut has = false;
|
||||
|
||||
@@ -2942,7 +2942,7 @@ impl MetaDeleteMarker {
|
||||
}
|
||||
|
||||
pub fn into_fileinfo(&self, volume: &str, path: &str, _all_parts: bool) -> Result<FileInfo> {
|
||||
let metadata = self
|
||||
let metadata: AHashMap<String, String> = self
|
||||
.meta_sys
|
||||
.clone()
|
||||
.into_iter()
|
||||
@@ -5500,10 +5500,10 @@ mod tests {
|
||||
/// entirely, silently dropping the whole legacy body on re-marshal.
|
||||
#[test]
|
||||
fn legacy_version_body_round_trips_through_encode() {
|
||||
let mut meta = HashMap::new();
|
||||
let mut meta = AHashMap::new();
|
||||
meta.insert("content-type".to_string(), "application/octet-stream".to_string());
|
||||
|
||||
let mut crc = HashMap::new();
|
||||
let mut crc = AHashMap::new();
|
||||
crc.insert("crc32c".to_string(), "deadbeef".to_string());
|
||||
|
||||
let legacy = MetaObjectV1 {
|
||||
|
||||
@@ -22,6 +22,12 @@ mod replication;
|
||||
|
||||
pub mod test_data;
|
||||
|
||||
/// High-performance HashMap type alias using ahash instead of SipHash.
|
||||
pub type AHashMap<K, V> = ahash::AHashMap<K, V>;
|
||||
|
||||
/// High-performance HashSet type alias using ahash.
|
||||
pub type AHashSet<K> = ahash::AHashSet<K>;
|
||||
|
||||
pub use error::*;
|
||||
pub use fileinfo::*;
|
||||
pub use filemeta::*;
|
||||
|
||||
@@ -14,8 +14,8 @@
|
||||
|
||||
use crate::filemeta::msgp_decode::MAX_MSGP_ELEMENT_SIZE;
|
||||
use crate::{
|
||||
Error, FileInfo, FileInfoOpts, FileInfoVersions, FileMeta, FileMetaShallowVersion, Result, VersionType, get_file_info,
|
||||
merge_file_meta_versions, merge_file_meta_versions_with_write_quorum,
|
||||
AHashMap, Error, FileInfo, FileInfoOpts, FileInfoVersions, FileMeta, FileMetaShallowVersion, Result, VersionType,
|
||||
get_file_info, merge_file_meta_versions, merge_file_meta_versions_with_write_quorum,
|
||||
};
|
||||
use arc_swap::ArcSwapOption;
|
||||
use rmp::Marker;
|
||||
@@ -1676,7 +1676,7 @@ mod tests {
|
||||
}
|
||||
|
||||
fn metacache_entry_with_mod_time(mod_time: OffsetDateTime, etag: &str) -> MetaCacheEntry {
|
||||
let mut metadata = HashMap::new();
|
||||
let mut metadata = AHashMap::new();
|
||||
metadata.insert("etag".to_string(), etag.to_string());
|
||||
|
||||
let mut meta = FileMeta::new();
|
||||
@@ -1705,7 +1705,7 @@ mod tests {
|
||||
data_blocks: usize,
|
||||
parity_blocks: usize,
|
||||
) -> MetaCacheEntry {
|
||||
let mut metadata = HashMap::new();
|
||||
let mut metadata = AHashMap::new();
|
||||
metadata.insert("etag".to_string(), etag.to_string());
|
||||
|
||||
let mut fi = FileInfo::new("object", data_blocks, parity_blocks);
|
||||
@@ -1730,7 +1730,7 @@ mod tests {
|
||||
fn metacache_entry_with_erasure_versions(versions: &[(OffsetDateTime, &str, usize, usize)]) -> MetaCacheEntry {
|
||||
let mut meta = FileMeta::new();
|
||||
for (idx, (mod_time, etag, data_blocks, parity_blocks)) in versions.iter().enumerate() {
|
||||
let mut metadata = HashMap::new();
|
||||
let mut metadata = AHashMap::new();
|
||||
metadata.insert("etag".to_string(), (*etag).to_string());
|
||||
|
||||
let mut fi = FileInfo::new("object", *data_blocks, *parity_blocks);
|
||||
@@ -1776,7 +1776,7 @@ mod tests {
|
||||
/// Build an entry holding a single object version with an explicit version id
|
||||
/// and mod_time, so a set of these can model DISJOINT per-disk version sets.
|
||||
fn metacache_entry_single_version(version_u128: u128, mod_time: OffsetDateTime, etag: &str) -> MetaCacheEntry {
|
||||
let mut metadata = HashMap::new();
|
||||
let mut metadata = AHashMap::new();
|
||||
metadata.insert("etag".to_string(), etag.to_string());
|
||||
|
||||
let mut fi = FileInfo::new("object", 4, 2);
|
||||
|
||||
@@ -81,7 +81,6 @@ pub enum StorageErrorCode {
|
||||
InsufficientWriteQuorum,
|
||||
PreconditionFailed,
|
||||
EntityTooSmall,
|
||||
EntityTooLarge,
|
||||
InvalidRangeSpec,
|
||||
NotModified,
|
||||
InvalidPartNumber,
|
||||
@@ -170,7 +169,6 @@ impl StorageErrorCode {
|
||||
Self::InsufficientWriteQuorum => 0x3A,
|
||||
Self::PreconditionFailed => 0x3B,
|
||||
Self::EntityTooSmall => 0x3C,
|
||||
Self::EntityTooLarge => 0x56,
|
||||
Self::InvalidRangeSpec => 0x3D,
|
||||
Self::NotModified => 0x3E,
|
||||
Self::InvalidPartNumber => 0x3F,
|
||||
@@ -259,7 +257,6 @@ impl StorageErrorCode {
|
||||
0x3A => Some(Self::InsufficientWriteQuorum),
|
||||
0x3B => Some(Self::PreconditionFailed),
|
||||
0x3C => Some(Self::EntityTooSmall),
|
||||
0x56 => Some(Self::EntityTooLarge),
|
||||
0x3D => Some(Self::InvalidRangeSpec),
|
||||
0x3E => Some(Self::NotModified),
|
||||
0x3F => Some(Self::InvalidPartNumber),
|
||||
@@ -353,7 +350,6 @@ mod tests {
|
||||
(StorageErrorCode::InsufficientWriteQuorum, 0x3A),
|
||||
(StorageErrorCode::PreconditionFailed, 0x3B),
|
||||
(StorageErrorCode::EntityTooSmall, 0x3C),
|
||||
(StorageErrorCode::EntityTooLarge, 0x56),
|
||||
(StorageErrorCode::InvalidRangeSpec, 0x3D),
|
||||
(StorageErrorCode::NotModified, 0x3E),
|
||||
(StorageErrorCode::InvalidPartNumber, 0x3F),
|
||||
|
||||
@@ -58,6 +58,9 @@ transform-stream = { workspace = true, optional = true }
|
||||
url = { workspace = true, optional = true }
|
||||
zstd = { workspace = true, optional = true }
|
||||
|
||||
# High-performance hashing
|
||||
ahash = { workspace = true, optional = true }
|
||||
|
||||
[dev-dependencies]
|
||||
criterion = { workspace = true, features = ["html_reports"] }
|
||||
tempfile = { workspace = true }
|
||||
|
||||
@@ -89,7 +89,7 @@ pub fn is_object_encryption_marker(key: &str) -> bool {
|
||||
}
|
||||
|
||||
/// Reads the logical object size recorded by encryption metadata.
|
||||
pub fn get_object_encryption_original_size(metadata: &std::collections::HashMap<String, String>) -> std::io::Result<Option<i64>> {
|
||||
pub fn get_object_encryption_original_size<S: std::hash::BuildHasher>(metadata: &std::collections::HashMap<String, String, S>) -> std::io::Result<Option<i64>> {
|
||||
let actual_size = super::get_str(metadata, super::SUFFIX_ACTUAL_SIZE);
|
||||
let size = get_case_insensitive(metadata, RUSTFS_ENCRYPTION_ORIGINAL_SIZE)
|
||||
.or_else(|| get_case_insensitive(metadata, SSEC_ORIGINAL_SIZE))
|
||||
@@ -103,7 +103,7 @@ pub fn get_object_encryption_original_size(metadata: &std::collections::HashMap<
|
||||
.map_err(|error| std::io::Error::other(format!("Failed to parse encryption original size: {error}")))
|
||||
}
|
||||
|
||||
fn get_case_insensitive<'a>(metadata: &'a std::collections::HashMap<String, String>, key: &str) -> Option<&'a str> {
|
||||
fn get_case_insensitive<'a, S: std::hash::BuildHasher>(metadata: &'a std::collections::HashMap<String, String, S>, key: &str) -> Option<&'a str> {
|
||||
metadata.get(key).map(String::as_str).or_else(|| {
|
||||
metadata
|
||||
.iter()
|
||||
|
||||
@@ -44,8 +44,6 @@ pub const SUFFIX_COMPRESSION: &str = "compression";
|
||||
pub const SUFFIX_REPLICATION_PRESERVE_CIPHERTEXT: &str = "replication-preserve-ciphertext";
|
||||
pub const SUFFIX_COMPRESSION_SIZE: &str = "compression-size";
|
||||
pub const SUFFIX_ACTUAL_SIZE: &str = "actual-size";
|
||||
/// Maximum logical object size for a capability-bound multipart upload.
|
||||
pub const SUFFIX_MAX_TOTAL_OBJECT_SIZE: &str = "max-total-object-size";
|
||||
pub const SUFFIX_ACTUAL_OBJECT_SIZE: &str = "actual-object-size";
|
||||
/// Used by replication; key stored with capital A
|
||||
pub const SUFFIX_ACTUAL_OBJECT_SIZE_CAP: &str = "Actual-Object-Size";
|
||||
@@ -184,13 +182,13 @@ pub fn internal_key_rustfs(suffix: &str) -> String {
|
||||
|
||||
// === String type (FileInfo.metadata, user_defined) ===
|
||||
|
||||
pub fn insert_str(map: &mut HashMap<String, String>, suffix: &str, value: String) {
|
||||
pub fn insert_str<S: std::hash::BuildHasher>(map: &mut HashMap<String, String, S>, suffix: &str, value: String) {
|
||||
let (k1, k2) = both_keys(suffix);
|
||||
map.insert(k1, value.clone());
|
||||
map.insert(k2, value);
|
||||
}
|
||||
|
||||
pub fn get_str(map: &HashMap<String, String>, suffix: &str) -> Option<String> {
|
||||
pub fn get_str<S: std::hash::BuildHasher>(map: &HashMap<String, String, S>, suffix: &str) -> Option<String> {
|
||||
if let Some(v) = with_internal_key(RUSTFS_INTERNAL_PREFIX, suffix, |k1| map.get(k1).cloned()) {
|
||||
return Some(v);
|
||||
}
|
||||
@@ -204,7 +202,7 @@ pub fn get_str(map: &HashMap<String, String>, suffix: &str) -> Option<String> {
|
||||
.map(|(_, value)| value.clone())
|
||||
}
|
||||
|
||||
fn get_consistent_value<'a, V: AsRef<[u8]>>(map: &'a HashMap<String, V>, suffix: &str) -> Option<&'a V> {
|
||||
fn get_consistent_value<'a, V: AsRef<[u8]>, S: std::hash::BuildHasher>(map: &'a HashMap<String, V, S>, suffix: &str) -> Option<&'a V> {
|
||||
let (rustfs_key, minio_key) = both_keys(suffix);
|
||||
let mut value = None;
|
||||
for (key, candidate) in map {
|
||||
@@ -222,11 +220,11 @@ fn get_consistent_value<'a, V: AsRef<[u8]>>(map: &'a HashMap<String, V>, suffix:
|
||||
/// Returns a non-empty value when every compatibility key present for `suffix` agrees.
|
||||
/// A single RustFS or MinIO key is accepted for backward compatibility; conflicting or empty
|
||||
/// values return `None` so callers at destructive boundaries can fail closed.
|
||||
pub fn get_consistent_str<'a>(map: &'a HashMap<String, String>, suffix: &str) -> Option<&'a str> {
|
||||
pub fn get_consistent_str<'a, S: std::hash::BuildHasher>(map: &'a HashMap<String, String, S>, suffix: &str) -> Option<&'a str> {
|
||||
get_consistent_value(map, suffix).map(String::as_str)
|
||||
}
|
||||
|
||||
pub fn contains_key_str(map: &HashMap<String, String>, suffix: &str) -> bool {
|
||||
pub fn contains_key_str<S: std::hash::BuildHasher>(map: &HashMap<String, String, S>, suffix: &str) -> bool {
|
||||
if with_internal_key(RUSTFS_INTERNAL_PREFIX, suffix, |k1| map.contains_key(k1)) {
|
||||
return true;
|
||||
}
|
||||
@@ -238,7 +236,7 @@ pub fn contains_key_str(map: &HashMap<String, String>, suffix: &str) -> bool {
|
||||
.any(|key| key.eq_ignore_ascii_case(&k1) || key.eq_ignore_ascii_case(&k2))
|
||||
}
|
||||
|
||||
pub fn remove_str(map: &mut HashMap<String, String>, suffix: &str) {
|
||||
pub fn remove_str<S: std::hash::BuildHasher>(map: &mut HashMap<String, String, S>, suffix: &str) {
|
||||
with_internal_key(RUSTFS_INTERNAL_PREFIX, suffix, |k1| map.remove(k1));
|
||||
with_internal_key(MINIO_INTERNAL_PREFIX, suffix, |k2| map.remove(k2));
|
||||
let (k1, k2) = both_keys(suffix);
|
||||
@@ -287,7 +285,7 @@ pub fn strip_internal_prefix_preserving_case(key: &str) -> Option<&str> {
|
||||
|
||||
/// Reads the bounded per-target delete-marker version map in one metadata scan.
|
||||
/// The boolean is set when matching metadata is malformed or compatibility keys disagree.
|
||||
pub fn target_delete_marker_versions(map: &HashMap<String, String>) -> (HashMap<String, String>, bool) {
|
||||
pub fn target_delete_marker_versions<S: std::hash::BuildHasher>(map: &HashMap<String, String, S>) -> (HashMap<String, String>, bool) {
|
||||
const MAX_ENTRIES: usize = 1_000;
|
||||
const MAX_ARN_LEN: usize = 1_024;
|
||||
const MAX_VERSION_ID_LEN: usize = 1_024;
|
||||
|
||||
@@ -81,7 +81,7 @@ Scope and exemptions:
|
||||
|
||||
- **SSE-KMS only.** SSE-S3 wraps its data key with a server-owned key the caller never names, and SSE-C never reaches KMS; both are exempt, matching AWS.
|
||||
- **The resolved key**, not the header. A bucket default encryption rule naming a KMS key is authorized the same way an explicit `x-amz-server-side-encryption-aws-kms-key-id` header is.
|
||||
- **Anonymous requests are denied.** An anonymous caller has no identity policy and therefore holds no `kms` grants, so under enforcement every anonymous read or write of an SSE-KMS object fails with `AccessDenied` — even when a bucket policy makes the bucket public. This matches AWS, where anonymous requests cannot use SSE-KMS objects at all, and it keeps the per-key gate meaningful: were anonymous requests exempt, any denied identity could bypass the gate on a public bucket by simply dropping its credentials. **A public bucket serving SSE-KMS objects is incompatible with enforcement** — serve public content unencrypted or under SSE-S3 instead. With enforcement off (the default), anonymous access to SSE-KMS objects remains governed by bucket policy alone. The server warns once per process when it first denies an anonymous request; per-request denials appear on audit entries (`kmsOutcome=failure`, `kmsErrorClass=access_denied`, empty requester identity) and at debug level.
|
||||
- **Anonymous requests are exempt.** They have no identity policy to evaluate, and denying them would break public buckets holding SSE-KMS objects. They remain governed by bucket policy.
|
||||
- **Internal work is exempt.** Replication, lifecycle transitions, healing and the scanner run as the system principal.
|
||||
- **Authorization runs before key state is checked**, so a denial cannot be used to probe whether a key exists, is disabled, or is pending deletion. The response is always `AccessDenied`.
|
||||
- **Multipart uploads are authorized at create time**, where the session data key is generated. Part uploads and completion reuse that envelope and are not re-authorized against the destination key.
|
||||
@@ -89,13 +89,13 @@ Scope and exemptions:
|
||||
|
||||
## Migration
|
||||
|
||||
Data-path enforcement is **off by default** because it changes the outcome of requests that succeed today: an identity holding only `s3:PutObject` can currently encrypt under any key. Turning it on without preparing policies will produce `AccessDenied` on working workloads.
|
||||
Data-path enforcement is **off by default in this release** because it changes the outcome of requests that succeed today: an identity holding only `s3:PutObject` can currently encrypt under any key. Turning it on without preparing policies will produce `AccessDenied` on working workloads.
|
||||
|
||||
```bash
|
||||
RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY=true
|
||||
```
|
||||
|
||||
The server logs the configured mode once at startup, and warns while enforcement is off. Enforcement stays opt-in: there is no roadmap to flip the default.
|
||||
The server logs the configured mode once at startup, and warns while enforcement is off. A later release defaults it to enabled.
|
||||
|
||||
Recommended sequence:
|
||||
|
||||
|
||||
@@ -1,55 +0,0 @@
|
||||
# Presigned multipart total-size limit
|
||||
|
||||
RustFS V2 supports an optional capability on a signed or SigV4-presigned
|
||||
`CreateMultipartUpload` request:
|
||||
|
||||
```text
|
||||
x-rustfs-max-total-object-size=<unsigned 64-bit integer>
|
||||
```
|
||||
|
||||
The backend must include the parameter before calculating the SigV4
|
||||
signature. It is part of the canonical query and cannot be added, removed, or
|
||||
changed by the browser. RustFS stores the verified limit in the multipart
|
||||
upload session and applies it to every `UploadPart` and to
|
||||
`CompleteMultipartUpload`.
|
||||
|
||||
Backend pseudocode (the custom query must be present before signing):
|
||||
|
||||
```text
|
||||
uri = "/photos/archive.zip?uploads"
|
||||
uri += "&x-rustfs-max-total-object-size=104857600"
|
||||
presigned_url = sigv4_presign("POST", uri, credentials)
|
||||
# Return presigned_url to the browser. Never append the parameter afterwards.
|
||||
```
|
||||
|
||||
The resulting flow is:
|
||||
|
||||
1. The backend signs `CreateMultipartUpload?...&x-rustfs-max-total-object-size=104857600`.
|
||||
2. RustFS verifies the SigV4 request and persists the limit with the upload ID.
|
||||
3. The browser uploads parts using the returned upload ID.
|
||||
4. RustFS rejects a part whose declared logical size would exceed the remaining
|
||||
budget and rejects completion if the server-side part metadata exceeds the
|
||||
limit.
|
||||
|
||||
The limit is measured in logical object bytes (`actual_size`), not erasure,
|
||||
encryption, or compression bytes. Replacing an existing part uses replacement
|
||||
semantics: the old part size is removed before the new part size is admitted.
|
||||
Unknown-length parts are rejected for capped sessions rather than buffered
|
||||
without a bound. Capped parts are admitted under an upload-wide write lock
|
||||
before temporary shards are created and use a per-upload staging permit to
|
||||
bound local in-flight data. The distributed lock is released while the body is
|
||||
read and reacquired for the final check/rename, so Complete and Abort are not
|
||||
blocked behind a slow upload. The normal request-body stall timeout releases
|
||||
the staging permit when a client stops sending.
|
||||
|
||||
The parameter is accepted only on `CreateMultipartUpload`. Supplying it on
|
||||
`UploadPart`, `CompleteMultipartUpload`, `AbortMultipartUpload`, listing, or
|
||||
copy operations returns `InvalidRequest`; those requests use the persisted
|
||||
session state. A multipart upload created without this parameter remains
|
||||
unlimited for backward compatibility. The V1 single-request capability
|
||||
(`x-rustfs-max-content-length`) is independent and is not a multipart limit.
|
||||
|
||||
Because enforcement happens in the multipart data plane, every node that may
|
||||
receive requests for a capped upload must run the V2 implementation. During a
|
||||
rolling upgrade, route capped uploads only to upgraded nodes; older nodes treat
|
||||
the internal metadata as unknown and cannot enforce the limit.
|
||||
@@ -119,11 +119,6 @@ pull-request gate.
|
||||
Use an exact preview tag for end-to-end release rehearsal. Manual dispatches
|
||||
are backfill/debug paths and do not prove the automatic `workflow_run` chain.
|
||||
|
||||
A preview Release is internal validation state, not a deliverable: after the
|
||||
final tag's release is published, `cleanup-preview-releases` deletes every
|
||||
`<target>-preview.<N>` Release for that target. The tags themselves are kept, so
|
||||
the validated commit stays traceable.
|
||||
|
||||
## Evidence requirements
|
||||
|
||||
A green check is useful only when it proves the intended behavior ran:
|
||||
|
||||
@@ -67,7 +67,6 @@ use super::storage_api::multipart_usecase::sse::{
|
||||
use super::storage_api::multipart_usecase::{
|
||||
StorageObjectInfo as ObjectInfo, StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader,
|
||||
};
|
||||
use crate::app::object::{guard_put_object_body_read_timeout, put_object_body_read_timeout};
|
||||
use crate::app::object_data_cache::{
|
||||
ObjectDataCacheAdapter, invalidate_object_data_cache_after_complete_multipart_success,
|
||||
invalidate_object_data_cache_before_mutation,
|
||||
@@ -79,11 +78,7 @@ use crate::app::object_usecase::{
|
||||
use crate::app::runtime_sources::{
|
||||
AppContext, current_app_context, current_object_data_cache_for_context, current_object_store_handle_for_context,
|
||||
};
|
||||
use crate::auth::{
|
||||
VerifiedPresignedRequest, VerifiedSigV4Request, parse_presigned_multipart_max_total_object_size,
|
||||
reject_presigned_multipart_max_total_object_size_for_other_operation,
|
||||
reject_presigned_put_max_content_length_for_other_operation,
|
||||
};
|
||||
use crate::auth::{VerifiedPresignedRequest, reject_presigned_put_max_content_length_for_other_operation};
|
||||
use crate::capacity::record_capacity_write;
|
||||
use crate::error::ApiError;
|
||||
use crate::table_catalog;
|
||||
@@ -97,9 +92,8 @@ use rustfs_utils::CompressionAlgorithm;
|
||||
#[cfg(test)]
|
||||
use rustfs_utils::http::insert_header;
|
||||
use rustfs_utils::http::{
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE, SUFFIX_REPLICATION_PRESERVE_CIPHERTEXT, SUFFIX_REPLICATION_STATUS,
|
||||
SUFFIX_REPLICATION_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_REQUEST, contains_key_str, get_consistent_str, get_header,
|
||||
get_source_scheme,
|
||||
SUFFIX_REPLICATION_PRESERVE_CIPHERTEXT, SUFFIX_REPLICATION_STATUS, SUFFIX_REPLICATION_TIMESTAMP,
|
||||
SUFFIX_SOURCE_REPLICATION_REQUEST, contains_key_str, get_header, get_source_scheme,
|
||||
headers::{AMZ_DECODED_CONTENT_LENGTH, AMZ_OBJECT_TAGGING, AMZ_STORAGE_CLASS},
|
||||
insert_str,
|
||||
};
|
||||
@@ -114,7 +108,6 @@ use s3s::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::str::FromStr;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
use tokio::sync::RwLock;
|
||||
use tokio_util::io::StreamReader;
|
||||
use tracing::{instrument, warn};
|
||||
@@ -233,22 +226,6 @@ fn create_multipart_upload_metadata(
|
||||
metadata
|
||||
}
|
||||
|
||||
fn multipart_max_total_object_size(metadata: &HashMap<String, String>) -> S3Result<Option<u64>> {
|
||||
if !contains_key_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE) {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
let value = get_consistent_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE).ok_or_else(|| {
|
||||
S3Error::with_message(
|
||||
S3ErrorCode::InvalidRequest,
|
||||
"multipart size capability metadata is missing or inconsistent".to_string(),
|
||||
)
|
||||
})?;
|
||||
value.parse::<u64>().map(Some).map_err(|_| {
|
||||
S3Error::with_message(S3ErrorCode::InvalidRequest, "multipart size capability metadata is invalid".to_string())
|
||||
})
|
||||
}
|
||||
|
||||
/// A multipart session advertises disk compression only when the staged-rollout
|
||||
/// switch (`RUSTFS_COMPRESSION_MULTIPART_ENABLED`) is on, the object key/headers
|
||||
/// qualify, AND the session is not an SSE-C ciphertext-passthrough replication
|
||||
@@ -421,11 +398,6 @@ impl DefaultMultipartUsecase {
|
||||
&self,
|
||||
req: S3Request<AbortMultipartUploadInput>,
|
||||
) -> S3Result<S3Response<AbortMultipartUploadOutput>> {
|
||||
reject_presigned_multipart_max_total_object_size_for_other_operation(
|
||||
&req.headers,
|
||||
req.uri.query(),
|
||||
req.extensions.get::<VerifiedSigV4Request>().is_some(),
|
||||
)?;
|
||||
reject_presigned_put_max_content_length_for_other_operation(
|
||||
&req.headers,
|
||||
req.uri.query(),
|
||||
@@ -472,11 +444,6 @@ impl DefaultMultipartUsecase {
|
||||
&self,
|
||||
req: S3Request<CompleteMultipartUploadInput>,
|
||||
) -> S3Result<S3Response<CompleteMultipartUploadOutput>> {
|
||||
reject_presigned_multipart_max_total_object_size_for_other_operation(
|
||||
&req.headers,
|
||||
req.uri.query(),
|
||||
req.extensions.get::<VerifiedSigV4Request>().is_some(),
|
||||
)?;
|
||||
reject_presigned_put_max_content_length_for_other_operation(
|
||||
&req.headers,
|
||||
req.uri.query(),
|
||||
@@ -785,11 +752,6 @@ impl DefaultMultipartUsecase {
|
||||
&self,
|
||||
req: S3Request<CreateMultipartUploadInput>,
|
||||
) -> S3Result<S3Response<CreateMultipartUploadOutput>> {
|
||||
let multipart_max_total_object_size = parse_presigned_multipart_max_total_object_size(
|
||||
&req.headers,
|
||||
req.uri.query(),
|
||||
req.extensions.get::<VerifiedSigV4Request>().is_some(),
|
||||
)?;
|
||||
reject_presigned_put_max_content_length_for_other_operation(
|
||||
&req.headers,
|
||||
req.uri.query(),
|
||||
@@ -845,9 +807,6 @@ impl DefaultMultipartUsecase {
|
||||
)?;
|
||||
|
||||
let mut metadata = create_multipart_upload_metadata(input_metadata, &req.headers, tagging, storage_class.as_ref());
|
||||
if let Some(limit) = multipart_max_total_object_size {
|
||||
insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, limit.to_string());
|
||||
}
|
||||
|
||||
let has_explicit_object_lock_retention = object_lock_mode.is_some()
|
||||
|| object_lock_retain_until_date.is_some()
|
||||
@@ -1019,11 +978,6 @@ impl DefaultMultipartUsecase {
|
||||
#[instrument(level = "debug", skip(self, req))]
|
||||
#[hotpath::measure(impl_type = "MultipartUsecase")]
|
||||
pub async fn execute_upload_part(&self, req: S3Request<UploadPartInput>) -> S3Result<S3Response<UploadPartOutput>> {
|
||||
reject_presigned_multipart_max_total_object_size_for_other_operation(
|
||||
&req.headers,
|
||||
req.uri.query(),
|
||||
req.extensions.get::<VerifiedSigV4Request>().is_some(),
|
||||
)?;
|
||||
reject_presigned_put_max_content_length_for_other_operation(
|
||||
&req.headers,
|
||||
req.uri.query(),
|
||||
@@ -1052,40 +1006,6 @@ impl DefaultMultipartUsecase {
|
||||
|
||||
let mut size = resolve_upload_part_size(&req.headers, content_length)?;
|
||||
let mut body_stream = body.ok_or_else(|| s3_error!(IncompleteBody))?;
|
||||
let Some(store) = self.object_store() else {
|
||||
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
|
||||
};
|
||||
let fi = store
|
||||
.get_multipart_info(&bucket, &key, &upload_id, &opts)
|
||||
.await
|
||||
.map_err(ApiError::from)?;
|
||||
let max_total_object_size = multipart_max_total_object_size(&fi.user_defined)?;
|
||||
if max_total_object_size.is_some() && size.is_some_and(|size| size < 0) {
|
||||
return Err(S3Error::new(S3ErrorCode::UnexpectedContent));
|
||||
}
|
||||
if max_total_object_size.is_some() && size.is_none() {
|
||||
return Err(S3Error::new(S3ErrorCode::UnexpectedContent));
|
||||
}
|
||||
if let (Some(limit), Some(size)) = (max_total_object_size, size)
|
||||
&& u64::try_from(size).is_ok_and(|size| size > limit)
|
||||
{
|
||||
return Err(S3Error::new(S3ErrorCode::EntityTooLarge));
|
||||
}
|
||||
if max_total_object_size.is_some() {
|
||||
let request_id = req
|
||||
.extensions
|
||||
.get::<super::storage_api::multipart_usecase::request_context::RequestContext>()
|
||||
.map(|ctx| ctx.request_id.clone())
|
||||
.unwrap_or_default();
|
||||
body_stream = guard_put_object_body_read_timeout(
|
||||
body_stream,
|
||||
&bucket,
|
||||
&key,
|
||||
&request_id,
|
||||
content_length,
|
||||
put_object_body_read_timeout().max(Duration::from_secs(rustfs_config::DEFAULT_HTTP_REQUEST_BODY_READ_TIMEOUT)),
|
||||
);
|
||||
}
|
||||
|
||||
if size.is_none() {
|
||||
let mut total = 0i64;
|
||||
@@ -1106,6 +1026,16 @@ impl DefaultMultipartUsecase {
|
||||
body_stream = StreamingBlob::wrap(stream);
|
||||
}
|
||||
|
||||
// Get multipart info early to check if managed encryption will be applied
|
||||
let Some(store) = self.object_store() else {
|
||||
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
|
||||
};
|
||||
|
||||
let fi = store
|
||||
.get_multipart_info(&bucket, &key, &upload_id, &opts)
|
||||
.await
|
||||
.map_err(ApiError::from)?;
|
||||
|
||||
let mut size = size.ok_or_else(|| s3_error!(UnexpectedContent))?;
|
||||
let ingress_stage_start = rustfs_io_metrics::put_stage_metrics_enabled().then(std::time::Instant::now);
|
||||
|
||||
@@ -1320,11 +1250,6 @@ impl DefaultMultipartUsecase {
|
||||
&self,
|
||||
req: S3Request<ListMultipartUploadsInput>,
|
||||
) -> S3Result<S3Response<ListMultipartUploadsOutput>> {
|
||||
reject_presigned_multipart_max_total_object_size_for_other_operation(
|
||||
&req.headers,
|
||||
req.uri.query(),
|
||||
req.extensions.get::<VerifiedSigV4Request>().is_some(),
|
||||
)?;
|
||||
reject_presigned_put_max_content_length_for_other_operation(
|
||||
&req.headers,
|
||||
req.uri.query(),
|
||||
@@ -1377,11 +1302,6 @@ impl DefaultMultipartUsecase {
|
||||
}
|
||||
|
||||
pub async fn execute_list_parts(&self, req: S3Request<ListPartsInput>) -> S3Result<S3Response<ListPartsOutput>> {
|
||||
reject_presigned_multipart_max_total_object_size_for_other_operation(
|
||||
&req.headers,
|
||||
req.uri.query(),
|
||||
req.extensions.get::<VerifiedSigV4Request>().is_some(),
|
||||
)?;
|
||||
reject_presigned_put_max_content_length_for_other_operation(
|
||||
&req.headers,
|
||||
req.uri.query(),
|
||||
@@ -1418,11 +1338,6 @@ impl DefaultMultipartUsecase {
|
||||
&self,
|
||||
req: S3Request<UploadPartCopyInput>,
|
||||
) -> S3Result<S3Response<UploadPartCopyOutput>> {
|
||||
reject_presigned_multipart_max_total_object_size_for_other_operation(
|
||||
&req.headers,
|
||||
req.uri.query(),
|
||||
req.extensions.get::<VerifiedSigV4Request>().is_some(),
|
||||
)?;
|
||||
reject_presigned_put_max_content_length_for_other_operation(
|
||||
&req.headers,
|
||||
req.uri.query(),
|
||||
@@ -1526,7 +1441,6 @@ impl DefaultMultipartUsecase {
|
||||
.get_multipart_info(&bucket, &key, &upload_id, &dst_opts)
|
||||
.await
|
||||
.map_err(ApiError::from)?;
|
||||
let destination_size_limit = multipart_max_total_object_size(&mp_info.user_defined)?;
|
||||
EncryptionRequest {
|
||||
bucket: &bucket,
|
||||
key: &key,
|
||||
@@ -1609,25 +1523,19 @@ impl DefaultMultipartUsecase {
|
||||
return Err(s3_error!(PreconditionFailed));
|
||||
}
|
||||
|
||||
let source_logical_size = match src_info.get_actual_size() {
|
||||
Ok(size) if size >= 0 => size,
|
||||
Ok(_) | Err(_) if destination_size_limit.is_some() => {
|
||||
return Err(S3Error::new(S3ErrorCode::UnexpectedContent));
|
||||
}
|
||||
Ok(_) | Err(_) => src_info.size,
|
||||
};
|
||||
|
||||
let (_start_offset, length) = if let Some(ref range_spec) = rs {
|
||||
// Copy-source ranges are expressed over the logical plaintext object.
|
||||
// Encrypted (and compressed) objects have a larger or smaller physical
|
||||
// representation, so validating against `size` rejects valid later parts.
|
||||
validate_copy_source_range_not_exceeds(range_spec, source_logical_size)?;
|
||||
let validation_size = src_info.get_actual_size().unwrap_or(src_info.size);
|
||||
|
||||
validate_copy_source_range_not_exceeds(range_spec, validation_size)?;
|
||||
|
||||
range_spec
|
||||
.get_offset_length(source_logical_size)
|
||||
.get_offset_length(validation_size)
|
||||
.map_err(|e| S3Error::with_message(S3ErrorCode::InvalidRange, e.to_string()))?
|
||||
} else {
|
||||
(0, source_logical_size)
|
||||
(0, src_info.size)
|
||||
};
|
||||
|
||||
let is_disk_compressed =
|
||||
@@ -2229,16 +2137,6 @@ mod tests {
|
||||
assert_eq!(metadata.get(AMZ_OBJECT_TAGGING), Some(&"project=rustfs".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multipart_max_total_object_size_reads_compatible_internal_metadata() {
|
||||
let mut metadata = HashMap::new();
|
||||
insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "104857600".to_string());
|
||||
assert_eq!(multipart_max_total_object_size(&metadata).unwrap(), Some(104_857_600));
|
||||
|
||||
metadata.insert("x-minio-internal-max-total-object-size".to_string(), "1".to_string());
|
||||
assert!(multipart_max_total_object_size(&metadata).is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn execute_complete_multipart_upload_rejects_missing_parts_payload() {
|
||||
let input = CompleteMultipartUploadInput::builder()
|
||||
|
||||
@@ -195,7 +195,6 @@ pub(crate) use self::delete::*;
|
||||
pub(crate) use self::extract::*;
|
||||
pub(crate) use self::get::*;
|
||||
use self::put::*;
|
||||
pub(crate) use self::put::{guard_put_object_body_read_timeout, put_object_body_read_timeout};
|
||||
pub(crate) use self::shared::*;
|
||||
#[cfg(test)]
|
||||
use self::test_support::*;
|
||||
|
||||
@@ -90,7 +90,7 @@ fn resolve_put_object_authoritative_size(headers: &HeaderMap, content_length: Op
|
||||
/// Returns `Duration::ZERO` when disabled (`RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0`),
|
||||
/// in which case [`guard_put_object_body_read_timeout`] passes the body through
|
||||
/// untouched.
|
||||
pub(crate) fn put_object_body_read_timeout() -> Duration {
|
||||
fn put_object_body_read_timeout() -> Duration {
|
||||
Duration::from_secs(rustfs_utils::get_env_u64(
|
||||
rustfs_config::ENV_HTTP_REQUEST_BODY_READ_TIMEOUT,
|
||||
rustfs_config::DEFAULT_HTTP_REQUEST_BODY_READ_TIMEOUT,
|
||||
@@ -260,7 +260,7 @@ impl ByteStream for RequestBodyReadTimeout {
|
||||
/// Wrap an incoming request body with [`RequestBodyReadTimeout`] unless the
|
||||
/// feature is disabled (`timeout == 0`), in which case the body is returned
|
||||
/// untouched. `remaining_length` is preserved via [`StreamingBlob::new`].
|
||||
pub(crate) fn guard_put_object_body_read_timeout(
|
||||
fn guard_put_object_body_read_timeout(
|
||||
body: StreamingBlob,
|
||||
bucket: &str,
|
||||
key: &str,
|
||||
@@ -962,7 +962,7 @@ impl DefaultObjectUsecase {
|
||||
rustfs_io_metrics::record_put_object_stage_duration_from("app_bucket_validate", bucket_validate_stage_start);
|
||||
|
||||
let put_admission = match get_concurrency_manager()
|
||||
.admit_put_object(size)
|
||||
.admit_put_object()
|
||||
.await
|
||||
.map_err(|_| s3_error!(InternalError, "foreground write admission closed"))?
|
||||
{
|
||||
|
||||
@@ -53,7 +53,6 @@ const EVENT_SESSION_TOKEN_EXTRACTION: &str = "session_token_extraction";
|
||||
|
||||
/// RustFS-specific query capability for a single presigned PutObject request.
|
||||
pub(crate) const RUSTFS_MAX_CONTENT_LENGTH_QUERY: &str = "x-rustfs-max-content-length";
|
||||
pub(crate) const RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY: &str = "x-rustfs-max-total-object-size";
|
||||
|
||||
/// Inserted by the S3 access boundary after the upstream verifier accepts a
|
||||
/// request as SigV4 presigned. Downstream capability parsing must require this
|
||||
@@ -61,9 +60,6 @@ pub(crate) const RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY: &str = "x-rustfs-max-total-
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub(crate) struct VerifiedPresignedRequest;
|
||||
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub(crate) struct VerifiedSigV4Request;
|
||||
|
||||
/// Performs constant-time string comparison to prevent timing attacks.
|
||||
///
|
||||
/// This function should be used when comparing sensitive values like passwords,
|
||||
@@ -1115,92 +1111,6 @@ pub(crate) fn reject_presigned_put_max_content_length_for_other_operation(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Parse the V2 multipart total-size capability after SigV4 authentication.
|
||||
/// Header-authenticated CreateMultipartUpload requests are accepted because the
|
||||
/// custom query is covered by the SigV4 canonical request; later multipart
|
||||
/// operations read the immutable value from the upload session metadata.
|
||||
pub(crate) fn parse_presigned_multipart_max_total_object_size(
|
||||
header: &HeaderMap,
|
||||
query: Option<&str>,
|
||||
verified_sigv4: bool,
|
||||
) -> S3Result<Option<u64>> {
|
||||
let Some(query) = query else {
|
||||
return Ok(None);
|
||||
};
|
||||
|
||||
let mut value = None;
|
||||
let mut decoded_query = Vec::new();
|
||||
for (name, candidate) in form_urlencoded::parse(query.as_bytes()) {
|
||||
decoded_query.push((name.to_string(), candidate.to_string()));
|
||||
if name == RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY {
|
||||
if value.is_some() {
|
||||
return Err(S3Error::with_message(
|
||||
S3ErrorCode::InvalidRequest,
|
||||
format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} must appear exactly once"),
|
||||
));
|
||||
}
|
||||
value = Some(candidate.into_owned());
|
||||
} else if name.eq_ignore_ascii_case(RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY) {
|
||||
return Err(S3Error::with_message(
|
||||
S3ErrorCode::InvalidRequest,
|
||||
format!("query parameter name must be exactly {RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY}"),
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
let Some(value) = value else {
|
||||
return Ok(None);
|
||||
};
|
||||
|
||||
let auth_type = get_request_auth_type_with_query(header, Some(query));
|
||||
let is_presigned = matches!(auth_type, AuthType::Presigned);
|
||||
let is_header_signed = matches!(auth_type, AuthType::Signed);
|
||||
let complete_presigned_query = [
|
||||
("x-amz-algorithm", "AWS4-HMAC-SHA256"),
|
||||
("x-amz-date", ""),
|
||||
("x-amz-expires", ""),
|
||||
("x-amz-signedheaders", ""),
|
||||
("x-amz-credential", ""),
|
||||
("x-amz-signature", ""),
|
||||
]
|
||||
.into_iter()
|
||||
.all(|(name, expected)| {
|
||||
decoded_query
|
||||
.iter()
|
||||
.find(|(candidate, _)| candidate.eq_ignore_ascii_case(name))
|
||||
.is_some_and(|(_, candidate)| !candidate.is_empty() && (expected.is_empty() || candidate == expected))
|
||||
});
|
||||
|
||||
let authenticated = verified_sigv4 && (is_header_signed || (is_presigned && complete_presigned_query));
|
||||
if !authenticated {
|
||||
return Err(S3Error::with_message(
|
||||
S3ErrorCode::InvalidRequest,
|
||||
format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} requires a verified SigV4 request"),
|
||||
));
|
||||
}
|
||||
|
||||
value.parse::<u64>().map(Some).map_err(|_| {
|
||||
S3Error::with_message(
|
||||
S3ErrorCode::InvalidRequest,
|
||||
format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} must be a non-negative 64-bit integer"),
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn reject_presigned_multipart_max_total_object_size_for_other_operation(
|
||||
header: &HeaderMap,
|
||||
query: Option<&str>,
|
||||
verified_sigv4: bool,
|
||||
) -> S3Result<()> {
|
||||
if parse_presigned_multipart_max_total_object_size(header, query, verified_sigv4)?.is_some() {
|
||||
return Err(S3Error::with_message(
|
||||
S3ErrorCode::InvalidRequest,
|
||||
format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} is only supported for CreateMultipartUpload"),
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
@@ -1906,52 +1816,6 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multipart_max_total_object_size_requires_signed_create_request() {
|
||||
let headers = HeaderMap::new();
|
||||
let signed_prefix = "X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20260827T000000Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Credential=test/20260827/us-east-1/s3/aws4_request&X-Amz-Signature=signature";
|
||||
let query = format!("{signed_prefix}&x-rustfs-max-total-object-size=104857600");
|
||||
|
||||
assert_eq!(
|
||||
parse_presigned_multipart_max_total_object_size(&headers, Some(&query), true).unwrap(),
|
||||
Some(104_857_600)
|
||||
);
|
||||
assert_eq!(
|
||||
reject_presigned_multipart_max_total_object_size_for_other_operation(&headers, Some(&query), true)
|
||||
.unwrap_err()
|
||||
.code(),
|
||||
&S3ErrorCode::InvalidRequest
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multipart_max_total_object_size_rejects_tampering_and_invalid_values() {
|
||||
let headers = HeaderMap::new();
|
||||
let signed_prefix = "X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20260827T000000Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Credential=test/20260827/us-east-1/s3/aws4_request&X-Amz-Signature=signature";
|
||||
for query in [
|
||||
"x-rustfs-max-total-object-size=1",
|
||||
"X-RustFS-Max-Total-Object-Size=1",
|
||||
"X-Amz-Algorithm=AWS4-HMAC-SHA256&x-rustfs-max-total-object-size=1&x-rustfs-max-total-object-size=2",
|
||||
"X-Amz-Algorithm=AWS4-HMAC-SHA256&x-rustfs-max-total-object-size=-1",
|
||||
"X-Amz-Algorithm=AWS4-HMAC-SHA256&x-rustfs-max-total-object-size=18446744073709551616",
|
||||
] {
|
||||
assert_eq!(
|
||||
parse_presigned_multipart_max_total_object_size(&headers, Some(query), true)
|
||||
.unwrap_err()
|
||||
.code(),
|
||||
&S3ErrorCode::InvalidRequest
|
||||
);
|
||||
}
|
||||
|
||||
let forged = format!("{signed_prefix}&x-rustfs-max-total-object-size=1");
|
||||
assert_eq!(
|
||||
parse_presigned_multipart_max_total_object_size(&headers, Some(&forged), false)
|
||||
.unwrap_err()
|
||||
.code(),
|
||||
&S3ErrorCode::InvalidRequest
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_credentials_is_expired() {
|
||||
let mut cred = create_test_credentials();
|
||||
|
||||
@@ -372,7 +372,6 @@ impl From<StorageError> for ApiError {
|
||||
StorageError::ObjectExistsAsDirectory(_, _) => S3ErrorCode::InvalidArgument,
|
||||
StorageError::InvalidPart(_, _, _) => S3ErrorCode::InvalidPart,
|
||||
StorageError::EntityTooSmall(_, _, _) => S3ErrorCode::EntityTooSmall,
|
||||
StorageError::EntityTooLarge(_, _) => S3ErrorCode::EntityTooLarge,
|
||||
StorageError::PreconditionFailed => S3ErrorCode::PreconditionFailed,
|
||||
StorageError::NotModified => S3ErrorCode::NotModified,
|
||||
StorageError::InvalidRangeSpec(_) => S3ErrorCode::InvalidRange,
|
||||
|
||||
@@ -16,10 +16,9 @@ use super::ObjectOptions;
|
||||
use super::ecfs::FS;
|
||||
use super::{ECStore, PolicySys, ReplicationStatusType, StorageError, get_lock_acquire_timeout, is_err_bucket_not_found};
|
||||
use crate::auth::{
|
||||
AuthType, RUSTFS_MAX_CONTENT_LENGTH_QUERY, RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY, VerifiedPresignedRequest,
|
||||
VerifiedSigV4Request, check_key_valid_with_context, get_condition_values_with_client_info,
|
||||
get_condition_values_with_query_and_client_info, get_request_auth_type_with_query, get_session_token,
|
||||
parse_presigned_multipart_max_total_object_size, parse_presigned_put_max_content_length,
|
||||
AuthType, RUSTFS_MAX_CONTENT_LENGTH_QUERY, VerifiedPresignedRequest, check_key_valid_with_context,
|
||||
get_condition_values_with_client_info, get_condition_values_with_query_and_client_info, get_request_auth_type_with_query,
|
||||
get_session_token, parse_presigned_put_max_content_length,
|
||||
};
|
||||
use crate::error::ApiError;
|
||||
use crate::license::license_check;
|
||||
@@ -1772,9 +1771,7 @@ impl S3Access for FS {
|
||||
|
||||
// Publish this server's context slot so downstream data-plane handlers
|
||||
// resolve the same store (backlog#1052 S6).
|
||||
let auth_type = get_request_auth_type_with_query(cx.headers(), cx.uri().query());
|
||||
let verified_presigned = matches!(auth_type, AuthType::Presigned);
|
||||
let verified_sigv4 = matches!(auth_type, AuthType::Presigned | AuthType::Signed);
|
||||
let verified_presigned = matches!(get_request_auth_type_with_query(cx.headers(), cx.uri().query()), AuthType::Presigned);
|
||||
{
|
||||
let ext = cx.extensions_mut();
|
||||
ext.insert(self.server_ctx().clone());
|
||||
@@ -1782,9 +1779,6 @@ impl S3Access for FS {
|
||||
if verified_presigned {
|
||||
ext.insert(VerifiedPresignedRequest);
|
||||
}
|
||||
if verified_sigv4 {
|
||||
ext.insert(VerifiedSigV4Request);
|
||||
}
|
||||
}
|
||||
|
||||
// The size capability is intentionally scoped to the single-object
|
||||
@@ -1799,14 +1793,6 @@ impl S3Access for FS {
|
||||
format!("{RUSTFS_MAX_CONTENT_LENGTH_QUERY} is only supported for presigned PutObject"),
|
||||
));
|
||||
}
|
||||
if parse_presigned_multipart_max_total_object_size(cx.headers(), cx.uri().query(), verified_sigv4)?.is_some()
|
||||
&& cx.s3_op().name() != "CreateMultipartUpload"
|
||||
{
|
||||
return Err(S3Error::with_message(
|
||||
S3ErrorCode::InvalidRequest,
|
||||
format!("{RUSTFS_MAX_TOTAL_OBJECT_SIZE_QUERY} is only supported for CreateMultipartUpload"),
|
||||
));
|
||||
}
|
||||
license_check().map_err(|er| match er.kind() {
|
||||
std::io::ErrorKind::PermissionDenied => s3_error!(AccessDenied, "{er}"),
|
||||
_ => {
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Concurrency manager for coordinating concurrent GetObject and PutObject requests.
|
||||
//! Concurrency manager for coordinating concurrent GetObject requests.
|
||||
|
||||
use super::io_schedule::{
|
||||
IoLoadLevel, IoLoadMetrics, IoPriority, IoPriorityQueue, IoPriorityQueueConfig, IoQueueStatus, IoSchedulerConfig, IoStrategy,
|
||||
@@ -34,8 +34,6 @@ use std::time::Duration;
|
||||
use tokio::sync::Semaphore;
|
||||
use tracing::debug;
|
||||
|
||||
const DERIVED_LARGE_PUT_ADMISSION_LIMIT_MAX: usize = 32;
|
||||
|
||||
/// Global concurrency manager instance
|
||||
pub(crate) static CONCURRENCY_MANAGER: LazyLock<ConcurrencyManager> = LazyLock::new(ConcurrencyManager::new);
|
||||
|
||||
@@ -67,8 +65,11 @@ pub struct ConcurrencyManager {
|
||||
bandwidth_monitor: Arc<Mutex<BandwidthMonitor>>,
|
||||
/// Metrics collector for I/O latency tracking (P50, P95, P99)
|
||||
metrics_collector: Arc<MetricsCollector>,
|
||||
/// Foreground PutObject admission policy, resolved once at startup.
|
||||
put_admission_policy: PutAdmissionPolicy,
|
||||
/// Experimental fixed-count foreground PutObject admission gate.
|
||||
put_admission_semaphore: Arc<Semaphore>,
|
||||
put_admission_enabled: bool,
|
||||
put_admission_limit: usize,
|
||||
put_admission_wait_timeout: Duration,
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for ConcurrencyManager {
|
||||
@@ -126,201 +127,10 @@ pub enum PutObjectAdmission {
|
||||
/// Request is admitted and must hold the permit until the store write
|
||||
/// returns or the request fails before mutation.
|
||||
Admitted(tokio::sync::OwnedSemaphorePermit),
|
||||
/// The selected foreground PUT admission gate stayed full until the configured wait timeout.
|
||||
/// The fixed-count gate stayed full until the configured wait timeout.
|
||||
Rejected,
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
struct PutAdmissionGate {
|
||||
semaphore: Arc<Semaphore>,
|
||||
limit: usize,
|
||||
wait_timeout: Duration,
|
||||
}
|
||||
|
||||
impl PutAdmissionGate {
|
||||
fn new(limit: usize, wait_timeout: Duration) -> Self {
|
||||
Self {
|
||||
semaphore: Arc::new(Semaphore::new(limit)),
|
||||
limit,
|
||||
wait_timeout,
|
||||
}
|
||||
}
|
||||
|
||||
fn active(&self) -> usize {
|
||||
self.limit.saturating_sub(self.semaphore.available_permits())
|
||||
}
|
||||
|
||||
async fn admit(&self) -> Result<PutObjectAdmission, tokio::sync::AcquireError> {
|
||||
if self.wait_timeout.is_zero() {
|
||||
return Ok(match self.semaphore.clone().try_acquire_owned() {
|
||||
Ok(permit) => PutObjectAdmission::Admitted(permit),
|
||||
Err(tokio::sync::TryAcquireError::NoPermits) => PutObjectAdmission::Rejected,
|
||||
Err(tokio::sync::TryAcquireError::Closed) => PutObjectAdmission::Rejected,
|
||||
});
|
||||
}
|
||||
|
||||
match tokio::time::timeout(self.wait_timeout, self.semaphore.clone().acquire_owned()).await {
|
||||
Ok(permit) => Ok(PutObjectAdmission::Admitted(permit?)),
|
||||
Err(_) => Ok(PutObjectAdmission::Rejected),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
enum PutAdmissionPolicy {
|
||||
/// Strict admission was explicitly enabled with limit `0`.
|
||||
Disabled,
|
||||
/// No hard PUT gate is configured; foreground write snapshots use the
|
||||
/// existing active request counter as a soft pressure signal.
|
||||
LegacyCounterOnly,
|
||||
/// Explicit all-PUT admission gate.
|
||||
Strict(PutAdmissionGate),
|
||||
/// Default large/unknown-size PUT admission gate.
|
||||
Large { gate: PutAdmissionGate, min_size_bytes: usize },
|
||||
}
|
||||
|
||||
impl PutAdmissionPolicy {
|
||||
fn from_env(max_disk_reads: usize) -> Self {
|
||||
let strict_enabled = rustfs_utils::get_env_bool(
|
||||
rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_ENABLE,
|
||||
rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE,
|
||||
);
|
||||
if strict_enabled {
|
||||
let strict_limit = rustfs_utils::get_env_usize(
|
||||
rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_LIMIT,
|
||||
rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_LIMIT,
|
||||
);
|
||||
let strict_wait_timeout = Duration::from_millis(rustfs_utils::get_env_u64(
|
||||
rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
|
||||
rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
|
||||
));
|
||||
return if strict_limit == 0 {
|
||||
Self::Disabled
|
||||
} else {
|
||||
Self::Strict(PutAdmissionGate::new(strict_limit, strict_wait_timeout))
|
||||
};
|
||||
}
|
||||
|
||||
let large_enabled = rustfs_utils::get_env_bool(
|
||||
rustfs_config::ENV_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE,
|
||||
rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE,
|
||||
);
|
||||
if !large_enabled {
|
||||
return Self::LegacyCounterOnly;
|
||||
}
|
||||
|
||||
let large_limit = derive_large_put_admission_limit(
|
||||
rustfs_utils::get_env_usize(
|
||||
rustfs_config::ENV_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT,
|
||||
rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT,
|
||||
),
|
||||
max_disk_reads,
|
||||
);
|
||||
let min_size_bytes = rustfs_utils::get_env_usize(
|
||||
rustfs_config::ENV_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES,
|
||||
rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES,
|
||||
);
|
||||
let wait_timeout = Duration::from_millis(rustfs_utils::get_env_u64(
|
||||
rustfs_config::ENV_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
|
||||
rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
|
||||
));
|
||||
|
||||
Self::Large {
|
||||
gate: PutAdmissionGate::new(large_limit, wait_timeout),
|
||||
min_size_bytes,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn strict_for_test(enabled: bool, limit: usize, wait_timeout: Duration) -> Self {
|
||||
if enabled {
|
||||
if limit == 0 {
|
||||
Self::Disabled
|
||||
} else {
|
||||
Self::Strict(PutAdmissionGate::new(limit, wait_timeout))
|
||||
}
|
||||
} else {
|
||||
Self::LegacyCounterOnly
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn large_for_test(enabled: bool, limit: usize, min_size_bytes: usize, wait_timeout: Duration) -> Self {
|
||||
if enabled && limit > 0 {
|
||||
Self::Large {
|
||||
gate: PutAdmissionGate::new(limit, wait_timeout),
|
||||
min_size_bytes,
|
||||
}
|
||||
} else {
|
||||
Self::LegacyCounterOnly
|
||||
}
|
||||
}
|
||||
|
||||
async fn admit(&self, size: i64) -> Result<PutObjectAdmission, tokio::sync::AcquireError> {
|
||||
match self {
|
||||
Self::Disabled | Self::LegacyCounterOnly => Ok(PutObjectAdmission::Disabled),
|
||||
Self::Strict(gate) => gate.admit().await,
|
||||
Self::Large { gate, min_size_bytes } if should_gate_large_put(size, *min_size_bytes) => gate.admit().await,
|
||||
Self::Large { .. } => Ok(PutObjectAdmission::Disabled),
|
||||
}
|
||||
}
|
||||
|
||||
fn snapshot(&self, legacy_limit: usize) -> WorkloadAdmissionSnapshot {
|
||||
match self {
|
||||
Self::Disabled => put_admission_snapshot(0, 0, None),
|
||||
Self::LegacyCounterOnly => put_admission_snapshot(PutObjectGuard::concurrent_count(), legacy_limit, None),
|
||||
Self::Strict(gate) => {
|
||||
put_admission_snapshot(gate.active(), gate.limit, Some("foreground write admission permits exhausted"))
|
||||
}
|
||||
Self::Large { gate, .. } => {
|
||||
put_admission_snapshot(gate.active(), gate.limit, Some("large foreground write admission permits exhausted"))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn put_admission_snapshot(active: usize, limit: usize, hard_gate_reason: Option<&'static str>) -> WorkloadAdmissionSnapshot {
|
||||
let state = if limit == 0 {
|
||||
AdmissionState::Disabled
|
||||
} else if active >= limit {
|
||||
AdmissionState::Saturated
|
||||
} else {
|
||||
AdmissionState::Open
|
||||
};
|
||||
|
||||
let admission =
|
||||
WorkloadAdmissionSnapshot::new(WorkloadClass::ForegroundWrite, state).with_counts(Some(active), None, Some(limit));
|
||||
|
||||
match state {
|
||||
AdmissionState::Disabled => admission.with_reason("foreground write admission disabled"),
|
||||
AdmissionState::Saturated => {
|
||||
admission.with_reason(hard_gate_reason.unwrap_or("foreground write concurrency reached local pressure limit"))
|
||||
}
|
||||
_ => admission,
|
||||
}
|
||||
}
|
||||
|
||||
fn derive_large_put_admission_limit(configured_limit: usize, max_disk_reads: usize) -> usize {
|
||||
if configured_limit > 0 {
|
||||
return configured_limit;
|
||||
}
|
||||
|
||||
let scheduler_base = if max_disk_reads == 0 {
|
||||
rustfs_config::DEFAULT_OBJECT_MAX_CONCURRENT_DISK_READS
|
||||
} else {
|
||||
max_disk_reads
|
||||
};
|
||||
scheduler_base.div_ceil(2).clamp(1, DERIVED_LARGE_PUT_ADMISSION_LIMIT_MAX)
|
||||
}
|
||||
|
||||
fn should_gate_large_put(size: i64, min_size_bytes: usize) -> bool {
|
||||
if min_size_bytes == 0 || size < 0 {
|
||||
return true;
|
||||
}
|
||||
|
||||
usize::try_from(size).is_ok_and(|size| size >= min_size_bytes)
|
||||
}
|
||||
|
||||
impl ConcurrencyManager {
|
||||
/// Create a new concurrency manager with default settings
|
||||
///
|
||||
@@ -367,7 +177,18 @@ impl ConcurrencyManager {
|
||||
// Initialize metrics collector for I/O latency tracking
|
||||
// Keep 1000 samples for P95/P99 calculation
|
||||
let metrics_collector = Arc::new(MetricsCollector::new(performance_metrics, 1000));
|
||||
let put_admission_policy = PutAdmissionPolicy::from_env(max_disk_reads);
|
||||
let put_admission_enabled = rustfs_utils::get_env_bool(
|
||||
rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_ENABLE,
|
||||
rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE,
|
||||
);
|
||||
let put_admission_limit = rustfs_utils::get_env_usize(
|
||||
rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_LIMIT,
|
||||
rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_LIMIT,
|
||||
);
|
||||
let put_admission_wait_timeout = Duration::from_millis(rustfs_utils::get_env_u64(
|
||||
rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
|
||||
rustfs_config::DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS,
|
||||
));
|
||||
|
||||
// Build queue config directly from scheduler config.
|
||||
let queue_config = IoPriorityQueueConfig::from_scheduler_config(&scheduler_config);
|
||||
@@ -383,7 +204,10 @@ impl ConcurrencyManager {
|
||||
pattern_detector,
|
||||
bandwidth_monitor,
|
||||
metrics_collector,
|
||||
put_admission_policy,
|
||||
put_admission_semaphore: Arc::new(Semaphore::new(if put_admission_enabled { put_admission_limit } else { 0 })),
|
||||
put_admission_enabled,
|
||||
put_admission_limit,
|
||||
put_admission_wait_timeout,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -410,19 +234,10 @@ impl ConcurrencyManager {
|
||||
#[cfg(test)]
|
||||
pub(crate) fn with_put_admission_for_test(enabled: bool, limit: usize, wait_timeout: Duration) -> Self {
|
||||
let mut manager = Self::new();
|
||||
manager.put_admission_policy = PutAdmissionPolicy::strict_for_test(enabled, limit, wait_timeout);
|
||||
manager
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn with_large_put_admission_for_test(
|
||||
enabled: bool,
|
||||
limit: usize,
|
||||
min_size_bytes: usize,
|
||||
wait_timeout: Duration,
|
||||
) -> Self {
|
||||
let mut manager = Self::new();
|
||||
manager.put_admission_policy = PutAdmissionPolicy::large_for_test(enabled, limit, min_size_bytes, wait_timeout);
|
||||
manager.put_admission_semaphore = Arc::new(Semaphore::new(if enabled { limit } else { 0 }));
|
||||
manager.put_admission_enabled = enabled;
|
||||
manager.put_admission_limit = limit;
|
||||
manager.put_admission_wait_timeout = wait_timeout;
|
||||
manager
|
||||
}
|
||||
|
||||
@@ -511,13 +326,30 @@ impl ConcurrencyManager {
|
||||
}
|
||||
}
|
||||
|
||||
/// Admit a foreground PutObject request under the configured write gate.
|
||||
/// Admit a foreground PutObject request under the experimental fixed-count gate.
|
||||
///
|
||||
/// The strict experimental gate applies to every PUT only when explicitly
|
||||
/// enabled. Otherwise the default-on large-object gate protects sustained
|
||||
/// erasure/RPC pressure while keeping small PUTs on the legacy path.
|
||||
pub async fn admit_put_object(&self, size: i64) -> Result<PutObjectAdmission, tokio::sync::AcquireError> {
|
||||
self.put_admission_policy.admit(size).await
|
||||
/// The default-off path returns [`PutObjectAdmission::Disabled`] without
|
||||
/// touching the semaphore, preserving legacy behavior. When enabled, the
|
||||
/// permit must be acquired before body ingest and held until the store write
|
||||
/// returns, so saturated foreground writes can fail with `SlowDown` before
|
||||
/// creating visible side effects.
|
||||
pub async fn admit_put_object(&self) -> Result<PutObjectAdmission, tokio::sync::AcquireError> {
|
||||
if !self.put_admission_enabled || self.put_admission_limit == 0 {
|
||||
return Ok(PutObjectAdmission::Disabled);
|
||||
}
|
||||
|
||||
if self.put_admission_wait_timeout.is_zero() {
|
||||
return Ok(match self.put_admission_semaphore.clone().try_acquire_owned() {
|
||||
Ok(permit) => PutObjectAdmission::Admitted(permit),
|
||||
Err(tokio::sync::TryAcquireError::NoPermits) => PutObjectAdmission::Rejected,
|
||||
Err(tokio::sync::TryAcquireError::Closed) => PutObjectAdmission::Rejected,
|
||||
});
|
||||
}
|
||||
|
||||
match tokio::time::timeout(self.put_admission_wait_timeout, self.put_admission_semaphore.clone().acquire_owned()).await {
|
||||
Ok(permit) => Ok(PutObjectAdmission::Admitted(permit?)),
|
||||
Err(_) => Ok(PutObjectAdmission::Rejected),
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================
|
||||
@@ -928,7 +760,35 @@ impl ConcurrencyManager {
|
||||
|
||||
/// Get a read-only workload admission snapshot for foreground writes.
|
||||
pub fn put_object_admission_snapshot(&self) -> WorkloadAdmissionSnapshot {
|
||||
self.put_admission_policy.snapshot(self.scheduler_config.max_concurrent_reads)
|
||||
let (active, limit, hard_gate_enabled) = if self.put_admission_enabled && self.put_admission_limit > 0 {
|
||||
(
|
||||
self.put_admission_limit
|
||||
.saturating_sub(self.put_admission_semaphore.available_permits()),
|
||||
self.put_admission_limit,
|
||||
true,
|
||||
)
|
||||
} else {
|
||||
(PutObjectGuard::concurrent_count(), self.scheduler_config.max_concurrent_reads, false)
|
||||
};
|
||||
let state = if limit == 0 {
|
||||
AdmissionState::Disabled
|
||||
} else if active >= limit {
|
||||
AdmissionState::Saturated
|
||||
} else {
|
||||
AdmissionState::Open
|
||||
};
|
||||
|
||||
let admission =
|
||||
WorkloadAdmissionSnapshot::new(WorkloadClass::ForegroundWrite, state).with_counts(Some(active), None, Some(limit));
|
||||
|
||||
match state {
|
||||
AdmissionState::Disabled => admission.with_reason("foreground write admission disabled"),
|
||||
AdmissionState::Saturated if hard_gate_enabled => {
|
||||
admission.with_reason("foreground write admission permits exhausted")
|
||||
}
|
||||
AdmissionState::Saturated => admission.with_reason("foreground write concurrency reached local pressure limit"),
|
||||
_ => admission,
|
||||
}
|
||||
}
|
||||
|
||||
/// Get a read-only workload admission registry snapshot for local storage concurrency.
|
||||
@@ -1002,7 +862,7 @@ impl Default for ConcurrencyManager {
|
||||
mod integration_tests {
|
||||
use super::super::io_schedule::{IoLoadLevel, IoPriority};
|
||||
use super::super::request_guard::GetObjectGuard;
|
||||
use super::{ConcurrencyManager, PutObjectAdmission, derive_large_put_admission_limit};
|
||||
use super::{ConcurrencyManager, PutObjectAdmission};
|
||||
use crate::storage::storage_api::concurrency_consumer::PutObjectGuard;
|
||||
use rustfs_concurrency::{AdmissionState, WorkloadAdmissionSnapshotProvider, WorkloadClass};
|
||||
use rustfs_io_core::io_profile::{AccessPattern, StorageMedia};
|
||||
@@ -1081,7 +941,7 @@ mod integration_tests {
|
||||
#[serial]
|
||||
async fn test_concurrency_manager_workload_admission_snapshot_tracks_put_requests() {
|
||||
crate::storage::concurrency::reset_active_put_requests();
|
||||
let manager = ConcurrencyManager::with_put_admission_for_test(false, 0, Duration::ZERO);
|
||||
let manager = ConcurrencyManager::new();
|
||||
let initial = manager.put_object_admission_snapshot();
|
||||
|
||||
assert_eq!(initial.class, WorkloadClass::ForegroundWrite);
|
||||
@@ -1105,42 +965,26 @@ mod integration_tests {
|
||||
let manager = ConcurrencyManager::with_put_admission_for_test(false, 1, Duration::ZERO);
|
||||
|
||||
let admission = manager
|
||||
.admit_put_object(1024)
|
||||
.admit_put_object()
|
||||
.await
|
||||
.expect("disabled put admission must not close");
|
||||
|
||||
assert!(matches!(admission, PutObjectAdmission::Disabled));
|
||||
assert_eq!(manager.put_admission_semaphore.available_permits(), 0);
|
||||
assert_eq!(manager.put_object_admission_snapshot().state, AdmissionState::Open);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_concurrency_manager_strict_put_admission_zero_limit_disables_large_gate() {
|
||||
let manager = ConcurrencyManager::with_put_admission_for_test(true, 0, Duration::ZERO);
|
||||
|
||||
let admission = manager
|
||||
.admit_put_object(32 * 1024 * 1024)
|
||||
.await
|
||||
.expect("strict zero-limit put admission must not close");
|
||||
|
||||
assert!(matches!(admission, PutObjectAdmission::Disabled));
|
||||
assert_eq!(manager.put_object_admission_snapshot().state, AdmissionState::Disabled);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_concurrency_manager_put_admission_rejects_when_limit_full() {
|
||||
let manager = ConcurrencyManager::with_put_admission_for_test(true, 1, Duration::ZERO);
|
||||
|
||||
let first = manager
|
||||
.admit_put_object(1024)
|
||||
.await
|
||||
.expect("first put admission should acquire");
|
||||
let first = manager.admit_put_object().await.expect("first put admission should acquire");
|
||||
assert!(matches!(first, PutObjectAdmission::Admitted(_)));
|
||||
assert_eq!(manager.put_object_admission_snapshot().state, AdmissionState::Saturated);
|
||||
|
||||
let second = manager
|
||||
.admit_put_object(1024)
|
||||
.admit_put_object()
|
||||
.await
|
||||
.expect("full put admission gate should reject, not close");
|
||||
assert!(matches!(second, PutObjectAdmission::Rejected));
|
||||
@@ -1151,14 +995,11 @@ mod integration_tests {
|
||||
async fn test_concurrency_manager_put_admission_reuses_released_permit() {
|
||||
let manager = ConcurrencyManager::with_put_admission_for_test(true, 1, Duration::ZERO);
|
||||
|
||||
let first = manager
|
||||
.admit_put_object(1024)
|
||||
.await
|
||||
.expect("first put admission should acquire");
|
||||
let first = manager.admit_put_object().await.expect("first put admission should acquire");
|
||||
drop(first);
|
||||
|
||||
let second = manager
|
||||
.admit_put_object(1024)
|
||||
.admit_put_object()
|
||||
.await
|
||||
.expect("released put admission permit should be reusable");
|
||||
assert!(matches!(second, PutObjectAdmission::Admitted(_)));
|
||||
@@ -1168,13 +1009,10 @@ mod integration_tests {
|
||||
#[serial]
|
||||
async fn test_concurrency_manager_put_admission_wait_timeout_rejects() {
|
||||
let manager = ConcurrencyManager::with_put_admission_for_test(true, 1, Duration::from_secs(5));
|
||||
let held = manager
|
||||
.admit_put_object(1024)
|
||||
.await
|
||||
.expect("first put admission should acquire");
|
||||
let held = manager.admit_put_object().await.expect("first put admission should acquire");
|
||||
let waiter_manager = manager.clone();
|
||||
|
||||
let waiter = tokio::spawn(async move { waiter_manager.admit_put_object(1024).await });
|
||||
let waiter = tokio::spawn(async move { waiter_manager.admit_put_object().await });
|
||||
tokio::task::yield_now().await;
|
||||
tokio::time::advance(Duration::from_secs(5)).await;
|
||||
|
||||
@@ -1186,85 +1024,6 @@ mod integration_tests {
|
||||
drop(held);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_concurrency_manager_large_put_admission_bypasses_small_puts() {
|
||||
let min_size = rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES;
|
||||
let manager = ConcurrencyManager::with_large_put_admission_for_test(true, 1, min_size, Duration::ZERO);
|
||||
|
||||
let held = manager
|
||||
.admit_put_object(min_size as i64)
|
||||
.await
|
||||
.expect("large put admission should acquire");
|
||||
assert!(matches!(held, PutObjectAdmission::Admitted(_)));
|
||||
|
||||
let small = manager
|
||||
.admit_put_object((min_size - 1) as i64)
|
||||
.await
|
||||
.expect("small put should bypass large admission");
|
||||
assert!(matches!(small, PutObjectAdmission::Disabled));
|
||||
|
||||
let large = manager
|
||||
.admit_put_object(min_size as i64)
|
||||
.await
|
||||
.expect("second large put should reject when the gate is full");
|
||||
assert!(matches!(large, PutObjectAdmission::Rejected));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_concurrency_manager_large_put_admission_gates_unknown_size() {
|
||||
let manager = ConcurrencyManager::with_large_put_admission_for_test(true, 1, 32 * 1024 * 1024, Duration::ZERO);
|
||||
|
||||
let held = manager
|
||||
.admit_put_object(-1)
|
||||
.await
|
||||
.expect("unknown-size put admission should acquire");
|
||||
assert!(matches!(held, PutObjectAdmission::Admitted(_)));
|
||||
|
||||
let second = manager
|
||||
.admit_put_object(-1)
|
||||
.await
|
||||
.expect("unknown-size put admission should reject when the gate is full");
|
||||
assert!(matches!(second, PutObjectAdmission::Rejected));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_concurrency_manager_large_put_snapshot_tracks_gate() {
|
||||
let manager = ConcurrencyManager::with_large_put_admission_for_test(true, 2, 32 * 1024 * 1024, Duration::ZERO);
|
||||
let first = manager
|
||||
.admit_put_object(32 * 1024 * 1024)
|
||||
.await
|
||||
.expect("first large put admission should acquire");
|
||||
let initial = manager.put_object_admission_snapshot();
|
||||
|
||||
assert_eq!(initial.class, WorkloadClass::ForegroundWrite);
|
||||
assert_eq!(initial.state, AdmissionState::Open);
|
||||
assert_eq!(initial.active, Some(1));
|
||||
assert_eq!(initial.limit, Some(2));
|
||||
|
||||
let second = manager
|
||||
.admit_put_object(32 * 1024 * 1024)
|
||||
.await
|
||||
.expect("second large put admission should acquire");
|
||||
let saturated = manager.put_object_admission_snapshot();
|
||||
|
||||
assert_eq!(saturated.state, AdmissionState::Saturated);
|
||||
assert_eq!(saturated.active, Some(2));
|
||||
assert_eq!(saturated.limit, Some(2));
|
||||
drop((first, second));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_concurrency_manager_derives_large_put_admission_limit_from_scheduler_cap() {
|
||||
assert_eq!(derive_large_put_admission_limit(7, 64), 7);
|
||||
assert_eq!(derive_large_put_admission_limit(0, 64), 32);
|
||||
assert_eq!(derive_large_put_admission_limit(0, 8), 4);
|
||||
assert_eq!(derive_large_put_admission_limit(0, 1), 1);
|
||||
assert_eq!(derive_large_put_admission_limit(0, 0), 32);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_concurrency_manager_workload_admission_registry_covers_required_classes() {
|
||||
|
||||
@@ -961,9 +961,9 @@ fn sse_kms_key_policy_enforced(principal: Option<&SseKmsPrincipal>) -> bool {
|
||||
|
||||
/// Report the configured SSE-KMS authorization mode once, at startup.
|
||||
///
|
||||
/// The disabled case warns rather than logs: while enforcement is off, any identity
|
||||
/// allowed to write an object can encrypt it under any key, and operators should hear
|
||||
/// about that even though disabled is the long-term default.
|
||||
/// The disabled case warns rather than logs: it is the compatibility default for this
|
||||
/// release only, and operators need the lead time to grant the kms actions before the
|
||||
/// default flips.
|
||||
pub(crate) fn log_sse_kms_key_policy_mode() {
|
||||
if sse_kms_key_policy_enforced(None) {
|
||||
tracing::info!(
|
||||
@@ -984,7 +984,7 @@ pub(crate) fn log_sse_kms_key_policy_mode() {
|
||||
"SSE-KMS requests are not authorized against the KMS key they name; any identity allowed to \
|
||||
write an object may encrypt it under any key, and any identity allowed to read it may have it \
|
||||
decrypted. Grant kms:GenerateDataKey and kms:Decrypt on the keys your workloads use, then set \
|
||||
{ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY}=true."
|
||||
{ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY}=true. A later release defaults this to enabled."
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1027,25 +1027,6 @@ async fn authorize_sse_kms_key(
|
||||
"Principal is not authorized for the KMS key resolved for this request"
|
||||
);
|
||||
|
||||
// One warn per process, not per request: anonymous denials are driven by
|
||||
// unauthenticated traffic, so a per-request warn would let anyone flood the
|
||||
// log. Per-request detail stays on the audit entry and the debug event above.
|
||||
if principal.account.is_empty() {
|
||||
static ANONYMOUS_DENIAL_WARNED: std::sync::Once = std::sync::Once::new();
|
||||
ANONYMOUS_DENIAL_WARNED.call_once(|| {
|
||||
tracing::warn!(
|
||||
component = LOG_COMPONENT_STORAGE,
|
||||
subsystem = LOG_SUBSYSTEM_SSE,
|
||||
event = "sse_kms_anonymous_key_authorization_denied",
|
||||
action = ?action,
|
||||
"Anonymous requests are being denied by SSE-KMS per-key authorization: anonymous \
|
||||
callers hold no kms grants, so a public bucket serving SSE-KMS objects is \
|
||||
incompatible with {ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY}=true. Reported once per \
|
||||
process; per-request denials are on audit entries and at debug level."
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
Err(ApiError {
|
||||
code: S3ErrorCode::AccessDenied,
|
||||
message: "Access Denied".to_string(),
|
||||
|
||||
@@ -140,14 +140,6 @@ done
|
||||
latest_guard="startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')"
|
||||
require_job_if "$build_workflow" "update-latest-version" " if: $latest_guard"
|
||||
require_line "$build_workflow" " needs: [ build-check, publish-release ]" "latest update must follow release publication"
|
||||
|
||||
# Preview releases are internal validation artifacts: once the deliverable
|
||||
# release is published they are deleted, while their tags stay behind.
|
||||
require_job_if "$build_workflow" "cleanup-preview-releases" " if: $latest_guard"
|
||||
require_line "$build_workflow" " gh release delete \"\$preview_tag\" --yes" "preview release cleanup after publication"
|
||||
require_line "$build_workflow" " | select(.tag_name | startswith(\$tag + \"-preview.\"))" "cleanup must match the target's own preview tags"
|
||||
require_line "$build_workflow" " | select(.tag_name | ltrimstr(\$tag + \"-preview.\") | test(\"^[0-9]+\$\"))" "cleanup must match a numeric preview iteration"
|
||||
require_absent "$build_workflow" "--cleanup-tag" "preview tags must survive their release cleanup"
|
||||
require_line "$build_workflow" " TARGET_COMMITISH=\$(git rev-parse --verify \"refs/tags/\${TAG}^{commit}\")" "release target commit resolution"
|
||||
require_line "$build_workflow" " ./scripts/release/create_or_update_release.sh \\" "managed release creation"
|
||||
require_absent "$build_workflow" "git tag -l --format='%(contents)'" "annotated tag messages must not become release notes"
|
||||
|
||||
@@ -1,104 +0,0 @@
|
||||
# RustFS Heal Test
|
||||
|
||||
Node-outage heal test driven by
|
||||
[`scripts/test/rustfs_heal_test.sh`](rustfs_heal_test.sh), based on the
|
||||
Obsidian note "RustFS Heal 测试步骤". Uses the same 3-node test environment as
|
||||
the pool expansion test (`vm000 vm001 vm002`).
|
||||
|
||||
All status checks talk to the RustFS admin API directly (SigV4-signed,
|
||||
`jq` assertions), no `rc` required.
|
||||
|
||||
## What it does
|
||||
|
||||
1. Downloads the `.deb` package on all nodes (release tag or a direct URL such
|
||||
as the nightly/R2 package).
|
||||
2. Installs it, writes the 3x4 config
|
||||
(`http://rustfs-node{1...3}:9000/data/rustfs{1...4}/mnmd`), starts all
|
||||
three nodes simultaneously, verifies the cluster is up.
|
||||
3. Writes data with `warp` while monitoring disk usage on the surviving nodes
|
||||
(`df -B1G | grep /data/rustfs`):
|
||||
- when both surviving nodes reach `STOP_NODE_AT_GB` (default 15 GiB), stop
|
||||
the outage node (`vm002`, `OUTAGE_NODE_INDEX=2`);
|
||||
- keep writing until both surviving nodes reach `WARP_STOP_AT_GB`
|
||||
(default 40 GiB), then stop warp.
|
||||
4. Restarts the outage node.
|
||||
5. Starts cluster heal: `POST /rustfs/admin/v3/heal/` with body
|
||||
`{"recursive":true}` (retried, returns a `clientToken`).
|
||||
6. Monitors the heal task via `POST /rustfs/admin/v3/heal/?clientToken=<token>`
|
||||
until the summary is a terminal success (`finished`/`completed`),
|
||||
`objects_failed == 0`, **and** the outage node's disk usage reaches
|
||||
`HEAL_TARGET_GB` (default 40 GiB).
|
||||
7. Result analysis: heal stats (scanned/healed/failed), per-node disk usage,
|
||||
pass/fail verdict.
|
||||
|
||||
Success requires **both** the heal API completion (the server's scan/repair
|
||||
verdict) and the outage node's disk reaching the target.
|
||||
|
||||
## Self-hosted runner prerequisites
|
||||
|
||||
- Register the admin host (e.g. `heal`) as a runner with the
|
||||
`smoke-testing` label.
|
||||
- Install `jq`, `openssl`, `curl` and `warp` on the runner. `rc` is **not**
|
||||
required.
|
||||
- The runner user must be able to SSH to `vm000/vm001/vm002` without a
|
||||
password prompt; nodes need passwordless `sudo` for the SSH user and
|
||||
resolvable `rustfs-node*` hostnames.
|
||||
- Admin API credentials need the `admin:server-info`, `admin:heal` and
|
||||
`admin:rebalance` actions.
|
||||
|
||||
## Configuration
|
||||
|
||||
Same repository secrets/variables as the pool expansion workflow:
|
||||
|
||||
| Kind | Name | Purpose |
|
||||
| ------ | --------------------- | ---------------------------------------------- |
|
||||
| Secret | `RUSTFS_ACCESS_KEY` | RustFS access key (default `rustfs@test`) |
|
||||
| Secret | `RUSTFS_SECRET_KEY` | RustFS secret key (default `rustfs@test`) |
|
||||
| Var | `RUSTFS_API_ENDPOINT` | Admin API endpoint, e.g. `http://127.0.0.1:9000` (`RUSTFS_RC_ENDPOINT` fallback) |
|
||||
| Var | `RUSTFS_NODES` | `vm000 vm001 vm002` |
|
||||
| Var | `RUSTFS_SSH_USER` | `azureuser` |
|
||||
| Var | `RUSTFS_NIGHTLY_PACKAGE_URL` | Default nightly deb URL (defaults to the R2 `latest` alias) |
|
||||
|
||||
## Workflow inputs
|
||||
|
||||
| Input | Default | Meaning |
|
||||
| ---------------- | ------- | ----------------------------------------- |
|
||||
| `package_url` | nightly | Direct `.deb` URL; empty = latest nightly |
|
||||
| `stop_node_gb` | `15` | Stop outage node at N GiB on survivors |
|
||||
| `warp_stop_gb` | `40` | Stop warp at N GiB on survivors |
|
||||
| `heal_target_gb` | `40` | Outage node must reach N GiB after heal |
|
||||
| `cleanup_before` | `true` | Reset nodes before the test |
|
||||
| `cleanup_after` | `true` | Reset nodes after the test |
|
||||
|
||||
> ⚠️ `--reset` purges the `rustfs` package and deletes the data directories on
|
||||
> all nodes. Only run against a dedicated test environment.
|
||||
|
||||
## Manual usage
|
||||
|
||||
```bash
|
||||
./scripts/test/rustfs_heal_test.sh --all -y \
|
||||
--package-url https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb \
|
||||
--endpoint http://127.0.0.1:9000
|
||||
|
||||
./scripts/test/rustfs_heal_test.sh --steps 5,6,7
|
||||
./scripts/test/rustfs_heal_test.sh --reset -y
|
||||
```
|
||||
|
||||
## Known issues
|
||||
|
||||
- Nightly builds gate pool/rebalance activation on a live fleet capability
|
||||
proof (rustfs/backlog#2031); the script retries heal/rebalance starts and
|
||||
prints a hint when the signature appears.
|
||||
- The cluster-level `GET /rustfs/admin/v3/background-heal/status` aggregator
|
||||
returns 501 in the single-pool 3x4 topology (no notification system), so the
|
||||
script monitors the started heal task via its `clientToken` instead.
|
||||
- The heal task may report `progress: null` while running; the script logs
|
||||
this as evidence (rustfs/backlog#2035) rather than coercing it to zero, and
|
||||
reads the canonical camelCase progress fields
|
||||
(`objectsScanned`/`objectsHealed`/`objectsFailed`/`progressPercentage`) with
|
||||
a snake_case fallback.
|
||||
- The server-side per-task heal timeout defaults to 5 minutes; the script
|
||||
writes `RUSTFS_HEAL_TASK_TIMEOUT_SECS=21600` (6h) into the node config so a
|
||||
multi-tens-of-GiB heal can finish. The background scanner is disabled
|
||||
(`RUSTFS_HEAL_AUTO_HEAL_ENABLE=false`) so the explicit heal is the only
|
||||
repair mechanism and the outage effect stays observable.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -93,9 +93,6 @@ WARP_BUCKET="test-10mb"
|
||||
WARP_OBJ_SIZE="100MiB"
|
||||
WARP_CONCURRENT=32
|
||||
WARP_DURATION="5m"
|
||||
# Warp log path; empty = auto-created unique temp file (the runner user may
|
||||
# not be able to write a shared /tmp path owned by another user).
|
||||
WARP_LOG_FILE="${RUSTFS_WARP_LOG_FILE:-}"
|
||||
STORAGE_THRESHOLD=85 # stop writing when usage reaches N% (note suggests 80-85)
|
||||
POLL_INTERVAL=30 # status polling interval (seconds)
|
||||
|
||||
@@ -105,8 +102,6 @@ DECOMMISSION_TIMEOUT=86400
|
||||
SERVICE_TIMEOUT=300
|
||||
DECOMMISSION_RETRIES=3 # auto clear+retry attempts after a failed decommission
|
||||
DECOMMISSION_RETRY_DELAY=30 # delay between retries (seconds)
|
||||
REBALANCE_START_RETRIES=6 # rebalance start retries (fleet proof may take ~10-20s after a topology change)
|
||||
REBALANCE_START_RETRY_DELAY=20 # delay between rebalance start retries (seconds)
|
||||
|
||||
# Pool to decommission (zero-based; 0 in the note)
|
||||
DECOMMISSION_POOL_ID=0
|
||||
@@ -330,45 +325,9 @@ wait_service_active() {
|
||||
sleep 5
|
||||
waited=$((waited + 5))
|
||||
done
|
||||
diagnose_node_start_failure "${node}"
|
||||
die "${node}: timed out waiting for ${RUSTFS_SERVICE} (${SERVICE_TIMEOUT}s)"
|
||||
}
|
||||
|
||||
# Known server-side issues the test can hit. Format:
|
||||
# "<error signature>|<tracking>|<hint>"
|
||||
KNOWN_SERVER_ISSUES=(
|
||||
"pool activation requires a live fleet capability proof|rustfs/backlog#2031|server-side cold-start recovery is covered by this PR; if this appears, collect node journals and treat it as a regression"
|
||||
)
|
||||
|
||||
# Print a hint when $1 matches a known server-side issue signature.
|
||||
hint_server_issue() {
|
||||
local text="$1" entry sig tracking hint
|
||||
for entry in "${KNOWN_SERVER_ISSUES[@]}"; do
|
||||
sig="${entry%%|*}"
|
||||
tracking="${entry#*|}"
|
||||
hint="${tracking#*|}"
|
||||
tracking="${tracking%%|*}"
|
||||
if printf '%s' "${text}" | grep -qiF "${sig}"; then
|
||||
printf '\033[1;33m[KNOWN SERVER ISSUE]\033[0m %s (%s): %s\n' "${sig}" "${tracking}" "${hint}" >&2
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Fetch the journal tail from a node whose service failed to start and
|
||||
# annotate known server-side issues.
|
||||
diagnose_node_start_failure() {
|
||||
local node="$1" journal
|
||||
if ! journal="$(ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" \
|
||||
"SUDO=\"\"; [ \"\$(id -u)\" -ne 0 ] && SUDO=\"sudo -n\"; \${SUDO} journalctl -u ${RUSTFS_SERVICE} --no-pager -n 60 2>/dev/null || true")"; then
|
||||
journal="unable to collect journal (SSH command failed)"
|
||||
fi
|
||||
printf '%s\n' "--- ${node}: ${RUSTFS_SERVICE} journal (last 60 lines) ---" >&2
|
||||
printf '%s\n' "${journal}" >&2
|
||||
hint_server_issue "${journal}" || true
|
||||
}
|
||||
|
||||
# Generate the /etc/default/rustfs content
|
||||
rustfs_config_body() {
|
||||
local volumes="$1"
|
||||
@@ -447,13 +406,9 @@ service_action() {
|
||||
local action="$1" node="$2"
|
||||
log "${node}: systemctl ${action} ${RUSTFS_SERVICE}"
|
||||
if [ "${DRY_RUN}" -eq 1 ]; then return 0; fi
|
||||
if ! ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" \
|
||||
"if [ \"\$(id -u)\" -ne 0 ]; then sudo -n systemctl ${action} ${RUSTFS_SERVICE}; else systemctl ${action} ${RUSTFS_SERVICE}; fi"; then
|
||||
if [ "${action}" = "start" ]; then
|
||||
diagnose_node_start_failure "${node}"
|
||||
fi
|
||||
die "${node}: systemctl ${action} failed"
|
||||
fi
|
||||
ssh "${SSH_OPTS[@]}" "${SSH_USER}@${node}" \
|
||||
"if [ \"\$(id -u)\" -ne 0 ]; then sudo -n systemctl ${action} ${RUSTFS_SERVICE}; else systemctl ${action} ${RUSTFS_SERVICE}; fi" \
|
||||
|| die "${node}: systemctl ${action} failed"
|
||||
}
|
||||
|
||||
service_action_all() {
|
||||
@@ -632,29 +587,6 @@ wait_rebalance() {
|
||||
die "timed out waiting for rebalance (${REBALANCE_TIMEOUT}s)"
|
||||
}
|
||||
|
||||
# Start rebalance via the admin API. Nightly builds gate rebalance activation
|
||||
# on a live cross-pool fence fleet capability proof that is re-established
|
||||
# shortly after a pool joins, so retry a few times before failing.
|
||||
start_rebalance_with_retry() {
|
||||
local attempts="${REBALANCE_START_RETRIES}" delay="${REBALANCE_START_RETRY_DELAY}" attempt=1 body code id
|
||||
while :; do
|
||||
body="$(admin_api POST /rustfs/admin/v3/rebalance/start "")"
|
||||
code="$(admin_api_code)"
|
||||
if [ "${code}" = "200" ]; then
|
||||
id="$(printf '%s' "${body}" | jq -r '.id // empty')"
|
||||
log "rebalance started: id=${id}"
|
||||
return 0
|
||||
fi
|
||||
warn "rebalance start attempt ${attempt}/${attempts} failed (HTTP ${code}): ${body}"
|
||||
if [ "${attempt}" -ge "${attempts}" ]; then
|
||||
hint_server_issue "${body}" || true
|
||||
die "rebalance start failed after ${attempts} attempts (see last error above)"
|
||||
fi
|
||||
attempt=$((attempt + 1))
|
||||
sleep "${delay}"
|
||||
done
|
||||
}
|
||||
|
||||
# Print a detailed decommission failure/progress report for one pool
|
||||
print_decommission_detail() {
|
||||
local body="$1" pool_id="$2" label="$3"
|
||||
@@ -885,18 +817,16 @@ step4_write_data() {
|
||||
log "DRY-RUN: monitoring storage usage until ${STORAGE_THRESHOLD}%"
|
||||
return 0
|
||||
fi
|
||||
local warp_log warp_pid
|
||||
warp_log="${WARP_LOG_FILE:-$(mktemp "${TMPDIR:-/tmp}/rustfs-warp.XXXXXX.log")}"
|
||||
log "starting warp writes (background), log: ${warp_log}"
|
||||
log "starting warp writes (background)..."
|
||||
warp put --host "${API_ENDPOINT#http://}" \
|
||||
--bucket "${WARP_BUCKET}" \
|
||||
--access-key "${ACCESS_KEY}" \
|
||||
--secret-key "${SECRET_KEY}" \
|
||||
--obj.size "${WARP_OBJ_SIZE}" \
|
||||
--concurrent "${WARP_CONCURRENT}" \
|
||||
--noprefix --duration "${WARP_DURATION}" --noclear >"${warp_log}" 2>&1 &
|
||||
warp_pid=$!
|
||||
log "warp PID=${warp_pid}"
|
||||
--noprefix --duration "${WARP_DURATION}" --noclear >/tmp/rustfs-warp.log 2>&1 &
|
||||
local warp_pid=$!
|
||||
log "warp PID=${warp_pid}, log /tmp/rustfs-warp.log"
|
||||
trap 'kill "${warp_pid:-}" 2>/dev/null || true' EXIT
|
||||
monitor_storage "${STORAGE_THRESHOLD}" "${warp_pid}"
|
||||
kill "${warp_pid}" 2>/dev/null || true
|
||||
@@ -936,8 +866,12 @@ step5_expand_pool2() {
|
||||
step6_rebalance() {
|
||||
log "step 6: start data rebalance (admin API)"
|
||||
confirm "About to start rebalance (POST ${API_ENDPOINT}/rustfs/admin/v3/rebalance/start). Continue?"
|
||||
local body id
|
||||
if [ "${DRY_RUN}" -eq 0 ]; then
|
||||
start_rebalance_with_retry
|
||||
body="$(admin_api POST /rustfs/admin/v3/rebalance/start "")"
|
||||
[ "$(admin_api_code)" = "200" ] || die "rebalance start failed (HTTP $(admin_api_code)): ${body}"
|
||||
id="$(printf '%s' "${body}" | jq -r '.id // empty')"
|
||||
log "rebalance started: id=${id}"
|
||||
fi
|
||||
wait_rebalance
|
||||
}
|
||||
@@ -960,8 +894,12 @@ step7_expand_pool3() {
|
||||
step8_rebalance() {
|
||||
log "step 8: start data rebalance (admin API)"
|
||||
confirm "About to start rebalance (POST ${API_ENDPOINT}/rustfs/admin/v3/rebalance/start). Continue?"
|
||||
local body id
|
||||
if [ "${DRY_RUN}" -eq 0 ]; then
|
||||
start_rebalance_with_retry
|
||||
body="$(admin_api POST /rustfs/admin/v3/rebalance/start "")"
|
||||
[ "$(admin_api_code)" = "200" ] || die "rebalance start failed (HTTP $(admin_api_code)): ${body}"
|
||||
id="$(printf '%s' "${body}" | jq -r '.id // empty')"
|
||||
log "rebalance started: id=${id}"
|
||||
fi
|
||||
wait_rebalance
|
||||
}
|
||||
@@ -990,7 +928,6 @@ step9_decommission() {
|
||||
break
|
||||
fi
|
||||
if [ "${attempt}" -ge "${DECOMMISSION_RETRIES}" ]; then
|
||||
warn "if the source bucket has many objects and the tested version is 1.0.0-rc.3, this is the known metacache-listing decommission bug; remove the test bucket (rc rb --force rustfs/${WARP_BUCKET}) or lower --storage-threshold, then re-run step 9"
|
||||
die "pool ${DECOMMISSION_POOL_ID} still failed after ${attempt} attempts; investigate manually (POST ${API_ENDPOINT}/rustfs/admin/v3/pools/clear?by-id=true&pool=${DECOMMISSION_POOL_ID} to reset)"
|
||||
fi
|
||||
warn "attempt ${attempt} failed; clearing metadata and retrying in ${DECOMMISSION_RETRY_DELAY}s"
|
||||
@@ -1128,12 +1065,6 @@ main() {
|
||||
trap 'rm -f "${ADMIN_API_CODE_FILE}"' EXIT
|
||||
if [ -n "${LOG_FILE}" ]; then
|
||||
mkdir -p "$(dirname "${LOG_FILE}")"
|
||||
if ! touch "${LOG_FILE}" 2>/dev/null; then
|
||||
# A fixed /tmp path may be owned by another user (e.g. a previous root
|
||||
# run); fall back to a unique, always-writable temp file.
|
||||
LOG_FILE="$(mktemp "${TMPDIR:-/tmp}/rustfs-pool-test.XXXXXX.log")"
|
||||
warn "log file not writable; using ${LOG_FILE}"
|
||||
fi
|
||||
exec > >(tee -a "${LOG_FILE}") 2>&1
|
||||
fi
|
||||
if [ "${RESET}" -eq 1 ]; then
|
||||
|
||||
Reference in New Issue
Block a user