Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2308d91990 | ||
|
|
346388b63c | ||
|
|
a56439219f | ||
|
|
0fe41da688 | ||
|
|
0953f7e912 | ||
|
|
75cd3885f3 | ||
|
|
73c9dd4c9d | ||
|
|
2040f5aff9 | ||
|
|
5104be1d23 | ||
|
|
5ef8b1ce5c | ||
|
|
ce4eca40a6 | ||
|
|
86b6fecbb4 | ||
|
|
88b43f546f | ||
|
|
2092fbf465 | ||
|
|
ed66b0a04d | ||
|
|
847fbd2a8b | ||
|
|
7eddd1cf83 | ||
|
|
2437069114 | ||
|
|
3b87d61cbf | ||
|
|
eb6b617ca2 | ||
|
|
64705d7589 | ||
|
|
206ef7d086 | ||
|
|
b301834c6d | ||
|
|
028be4f604 | ||
|
|
6f9adb3ad0 | ||
|
|
19c7529d88 | ||
|
|
7951601ae8 | ||
|
|
b87ce6b183 | ||
|
|
f135583fee | ||
|
|
75e605fe87 | ||
|
|
d115f1cbd7 | ||
|
|
cfaf87360f | ||
|
|
876f60c1f4 | ||
|
|
488af5984c | ||
|
|
7136062c75 | ||
|
|
1585308f0f | ||
|
|
e388a3ff53 | ||
|
|
22741603f5 | ||
|
|
3c89c71f66 | ||
|
|
db57fabcbd | ||
|
|
03888bd266 | ||
|
|
a6a04b5faa | ||
|
|
18068eb7e5 | ||
|
|
921a48bd14 | ||
|
|
2e6511566e | ||
|
|
ec8abb19ab | ||
|
|
2a8be5566d | ||
|
|
bcbc58b6a0 | ||
|
|
28fa412a06 | ||
|
|
6f7a4ff060 | ||
|
|
8a57632bfd | ||
|
|
2eb4ddf4af | ||
|
|
e281ed2f6d | ||
|
|
2e6c820f53 | ||
|
|
d48dda5bdc | ||
|
|
44f3f0e73e | ||
|
|
152f110583 | ||
|
|
f1de19fc14 | ||
|
|
7c4e514ec9 | ||
|
|
13a2ae212e | ||
|
|
94a6da6e83 | ||
|
|
a199312e45 | ||
|
|
3420006762 | ||
|
|
c95b4f0820 | ||
|
|
d9080ae77f | ||
|
|
95c926dc79 |
@@ -6,7 +6,7 @@ description: "Run the end-to-end RustFS console gate, version bump, preview vali
|
||||
|
||||
This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
|
||||
|
||||
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
|
||||
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. That Release is temporary: `build.yml` deletes it automatically once the final tag's Release is published, so the Releases page ends up carrying deliverables only while the `-preview.N` tags stay behind as the traceability record. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
|
||||
|
||||
Pipeline shape:
|
||||
|
||||
@@ -19,6 +19,7 @@ check console main against its latest Release
|
||||
-> validate with latest rc client
|
||||
-> report preview acceptance results -> STOP for explicit human confirmation
|
||||
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
|
||||
-> CI deletes the <target>-preview.N Releases (tags kept)
|
||||
```
|
||||
|
||||
On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
|
||||
@@ -51,14 +52,16 @@ Rules:
|
||||
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
|
||||
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
|
||||
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
|
||||
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
|
||||
|
||||
## Hard rules
|
||||
|
||||
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
|
||||
- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
|
||||
- Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
|
||||
- Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
|
||||
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
|
||||
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
|
||||
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
|
||||
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
|
||||
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
|
||||
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
|
||||
- Completing preview acceptance does not authorize the final tag. After Phases 3–5 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
|
||||
@@ -230,6 +233,7 @@ git push origin "<target>"
|
||||
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
|
||||
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
|
||||
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
|
||||
- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
|
||||
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
|
||||
|
||||
## Output contract
|
||||
@@ -239,5 +243,5 @@ Always report:
|
||||
- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
|
||||
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
|
||||
- Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
|
||||
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
|
||||
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
|
||||
- Any deviation from this pipeline and why the user approved it.
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
sha256-darwin=d6aa36cfaae2c4d8590482c7e47138c5965b335b34a75f50d11ffc3366e9021e
|
||||
sha256-linux=c8315465f50c194faee36141cdbb1e15e59271e524d948564a69e2d5eb408f2a
|
||||
sha256-linux=96db8060fce98addda4f69092d297ca236bec4892d820617a26a261eedac61b0
|
||||
|
||||
@@ -1 +1 @@
|
||||
sha256=655a3f3c1d042e694339d15caba7580518320322d1bac0f09450b37e6c09e2e7
|
||||
sha256=8d5517f5f2fc32d561782dfccd51b7f746f5e25b2835e37e100c883f7f18777d
|
||||
|
||||
@@ -5,3 +5,4 @@ self-hosted-runner:
|
||||
- sm-standard-2
|
||||
- sm-standard-4
|
||||
- dind-sm-standard-2
|
||||
- smoke-testing
|
||||
|
||||
@@ -7,6 +7,11 @@
|
||||
{ "workflow": ".github/workflows/e2e-s3tests.yml", "max_age_hours": 192 },
|
||||
{ "workflow": ".github/workflows/fuzz.yml", "max_age_hours": 36 },
|
||||
{ "workflow": ".github/workflows/mint.yml", "max_age_hours": 192 },
|
||||
{
|
||||
"workflow": ".github/workflows/minio-interop.yml",
|
||||
"max_age_hours": 36,
|
||||
"never_ran_grace_until": "2026-09-08T00:00:00Z"
|
||||
},
|
||||
{ "workflow": ".github/workflows/nightly-gnu.yml", "max_age_hours": 36 },
|
||||
{ "workflow": ".github/workflows/performance-ab.yml", "max_age_hours": 36 },
|
||||
{
|
||||
|
||||
@@ -1033,6 +1033,55 @@ jobs:
|
||||
echo "🎉 Released $TAG successfully!"
|
||||
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
|
||||
|
||||
# Remove the internal preview releases once the deliverable release is live.
|
||||
# Only the Releases are deleted; the -preview.N tags stay so the validated
|
||||
# commit remains traceable.
|
||||
cleanup-preview-releases:
|
||||
name: Cleanup Preview Releases
|
||||
needs: [ build-check, publish-release ]
|
||||
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: write
|
||||
steps:
|
||||
- name: Delete preview releases for this target
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
TAG="${{ needs.build-check.outputs.version }}"
|
||||
RELEASES_JSON="${RUNNER_TEMP}/releases.json"
|
||||
|
||||
# Fetch before filtering: a failed listing must abort here instead of
|
||||
# looking like "nothing to clean up".
|
||||
gh api --paginate "repos/${GITHUB_REPOSITORY}/releases?per_page=100" > "$RELEASES_JSON"
|
||||
|
||||
# Match only <target>-preview.<digits>. String operations, not a
|
||||
# regex over the tag, so dots in the version cannot widen the match.
|
||||
DELETED=0
|
||||
while IFS= read -r preview_tag; do
|
||||
[[ -n "$preview_tag" ]] || continue
|
||||
echo "🧹 Deleting preview release $preview_tag (tag kept)"
|
||||
gh release delete "$preview_tag" --repo "${GITHUB_REPOSITORY}" --yes
|
||||
DELETED=$((DELETED + 1))
|
||||
done < <(
|
||||
jq -r --arg tag "$TAG" '
|
||||
.[]
|
||||
| select(.tag_name | startswith($tag + "-preview."))
|
||||
| select(.tag_name | ltrimstr($tag + "-preview.") | test("^[0-9]+$"))
|
||||
| .tag_name
|
||||
' "$RELEASES_JSON"
|
||||
)
|
||||
|
||||
if [[ "$DELETED" -eq 0 ]]; then
|
||||
echo "ℹ️ No preview releases to clean up for $TAG"
|
||||
else
|
||||
echo "✅ Removed $DELETED preview release(s) for $TAG"
|
||||
fi
|
||||
|
||||
alert-on-failure:
|
||||
name: Alert on scheduled failure
|
||||
needs: [build-check, prepare-platform-matrix, build-rustfs, build-summary]
|
||||
|
||||
@@ -20,27 +20,27 @@
|
||||
# each run with Docker and then runs the `#[ignore]` reader tests in
|
||||
# rustfs/src/storage/minio_generated_read_test.rs.
|
||||
#
|
||||
# Scope: end-to-end MinIO-to-RustFS SSE interop is NOT implemented yet. Both
|
||||
# envelope parsers reject MinIO's own wrapped-DEK shape — see
|
||||
# `is_data_key_envelope` in crates/kms/src/encryption/dek.rs and the
|
||||
# `deny_unknown_fields` `LocalSseDekEnvelope` in rustfs/src/storage/sse.rs — and
|
||||
# closing that gap is tracked in rustfs/backlog#1638. Treat this job as the
|
||||
# harness for #1638, not as standing evidence that a MinIO migration reads back.
|
||||
# Scope: MinIO-to-RustFS SSE read interop is implemented behind the `rio-v2`
|
||||
# feature for MinIO's builtin static-KMS deployments — SSE-S3 and SSE-KMS
|
||||
# (single- and multipart) since rustfs/rustfs#6191, SSE-C detection since the
|
||||
# rustfs/backlog#1638 D2 close-out. This job is the standing evidence: it
|
||||
# regenerates real MinIO backend trees and proves byte-identical plaintext
|
||||
# reconstruction. KES/MinKMS-backed MinIO objects remain unreadable by design
|
||||
# (their envelopes are sealed by the KES service, not by a key RustFS can
|
||||
# hold), and default RustFS builds do not include the read path — it is a
|
||||
# special-purpose migration capability, not a default-build feature.
|
||||
#
|
||||
# Runner: GitHub-hosted `ubuntu-latest`. It reliably ships Docker + Python,
|
||||
# unlike the self-hosted fleet, whose pods drift in Docker/pip availability
|
||||
# (see the infra note in e2e-s3tests.yml). Nightly + manual only.
|
||||
# DISABLED. This workflow is switched off in the repository's Actions settings
|
||||
# (state: disabled_manually) and does not run on any trigger, including its cron
|
||||
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
|
||||
# reading this file, which has already misled at least one audit — hence this
|
||||
# banner. Re-enabling is a UI action; anyone doing so should first check that the
|
||||
# workflow still matches the current CI layout. See rustfs/backlog#1603.
|
||||
#
|
||||
# While disabled, this workflow is deliberately absent from
|
||||
# .github/scheduled-validations.json — a disabled workflow can never satisfy the
|
||||
# freshness check. Whoever re-enables it must re-add the entry in the same
|
||||
# change so the freshness gate covers it again.
|
||||
# Enablement: this workflow was long disabled in the repository's Actions
|
||||
# settings (state: disabled_manually — a state that lives in GitHub's UI and is
|
||||
# invisible in this file). The change that updated this banner also re-added
|
||||
# the .github/scheduled-validations.json entry; both only make sense together
|
||||
# with re-enabling the workflow in the Actions settings. If it is ever disabled
|
||||
# again, remove the scheduled-validations entry in the same change — a disabled
|
||||
# workflow can never satisfy the freshness check. See rustfs/backlog#1603.
|
||||
#
|
||||
name: minio-interop
|
||||
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
name: RustFS Heal Test
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
package_url:
|
||||
description: 'Direct .deb URL (nightly/R2). Defaults to the latest nightly deb.'
|
||||
required: false
|
||||
type: string
|
||||
stop_node_gb:
|
||||
description: 'Stop the outage node when surviving nodes reach N GiB'
|
||||
required: false
|
||||
default: '15'
|
||||
warp_stop_gb:
|
||||
description: 'Stop warp when surviving nodes reach N GiB'
|
||||
required: false
|
||||
default: '40'
|
||||
cleanup_before:
|
||||
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
|
||||
type: boolean
|
||||
default: true
|
||||
cleanup_after:
|
||||
description: 'Reset the nodes after the test (DESTROYS test data/config)'
|
||||
type: boolean
|
||||
default: true
|
||||
workflow_run:
|
||||
# Run after the nightly build completes: the nightly deb is what the test
|
||||
# installs. Heal also runs inside the functional suite; this standalone
|
||||
# workflow enables manual single-suite runs as well.
|
||||
workflows: ["Nightly GNU Build"]
|
||||
types: [completed]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# Only one test at a time: both this and the pool-expansion workflow mutate
|
||||
# the same test environment, so they share one concurrency group.
|
||||
concurrency:
|
||||
group: rustfs-pool-expansion-test
|
||||
cancel-in-progress: false
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
|
||||
env:
|
||||
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
|
||||
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
|
||||
RUSTFS_API_ENDPOINT: ${{ secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }}
|
||||
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
|
||||
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
|
||||
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
|
||||
|
||||
jobs:
|
||||
heal-test:
|
||||
runs-on: smoke-testing
|
||||
timeout-minutes: 480
|
||||
# Run on manual dispatch, or when the nightly build completed successfully.
|
||||
# Skipped when nightly failed.
|
||||
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
|
||||
steps:
|
||||
- name: Checkout auto-testing scripts
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: main
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
- name: Show environment
|
||||
run: |
|
||||
uname -a
|
||||
jq --version
|
||||
openssl version
|
||||
warp --version || true
|
||||
df -h /data | tail -1
|
||||
|
||||
- name: Reset test environment (before)
|
||||
if: ${{ inputs.cleanup_before != 'false' }}
|
||||
run: |
|
||||
chmod +x auto-testing/rustfs_heal_test.sh
|
||||
./auto-testing/rustfs_heal_test.sh --reset -y
|
||||
|
||||
- name: Install RustFS package & start cluster
|
||||
run: |
|
||||
ARGS=(--steps "1,2" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Preflight checks
|
||||
run: |
|
||||
ARGS=(--preflight --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Run heal test (write -> outage -> heal -> verify)
|
||||
run: |
|
||||
./auto-testing/rustfs_heal_test.sh \
|
||||
--steps "3,4,5,6,7" -y \
|
||||
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
|
||||
--stop-node-gb "${{ inputs.stop_node_gb }}" \
|
||||
--warp-stop-gb "${{ inputs.warp_stop_gb }}" \
|
||||
--log-file /tmp/rustfs-heal-test.log
|
||||
|
||||
- name: Upload test logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
with:
|
||||
name: rustfs-heal-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-heal-test*.log
|
||||
/tmp/rustfs-warp.*.log
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Reset test environment (after)
|
||||
if: ${{ always() && inputs.cleanup_after != 'false' }}
|
||||
run: |
|
||||
./auto-testing/rustfs_heal_test.sh --reset -y
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: |
|
||||
echo "RustFS heal test failed"
|
||||
echo "Package source: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
|
||||
echo "See the uploaded log artifact for details."
|
||||
@@ -0,0 +1,206 @@
|
||||
name: RustFS Performance Test
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
package_url:
|
||||
description: 'Direct .deb URL (nightly/R2). Defaults to the latest nightly deb.'
|
||||
required: false
|
||||
type: string
|
||||
test_method:
|
||||
description: 'Benchmark method(s) to run (manual runs only; "all" = GET+PUT+MIXED)'
|
||||
type: choice
|
||||
options:
|
||||
- all
|
||||
- get
|
||||
- put
|
||||
- mixed
|
||||
default: 'all'
|
||||
object_size:
|
||||
description: 'Object size(s) to test (manual runs only; "all" = all 10 sizes)'
|
||||
type: choice
|
||||
options:
|
||||
- all
|
||||
- 1KiB
|
||||
- 4KiB
|
||||
- 16KiB
|
||||
- 128KiB
|
||||
- 1MiB
|
||||
- 4MiB
|
||||
- 8MiB
|
||||
- 16MiB
|
||||
- 32MiB
|
||||
- 64MiB
|
||||
default: 'all'
|
||||
warp_duration:
|
||||
description: 'warp duration per round (e.g. 5m, 30s)'
|
||||
required: false
|
||||
default: '5m'
|
||||
warp_concurrency:
|
||||
description: 'warp concurrency'
|
||||
required: false
|
||||
default: '64'
|
||||
cleanup_before:
|
||||
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
|
||||
type: boolean
|
||||
default: true
|
||||
cleanup_after:
|
||||
description: 'Reset the nodes after the test (DESTROYS test data/config)'
|
||||
type: boolean
|
||||
default: true
|
||||
workflow_run:
|
||||
# Run after the nightly build completes; the nightly deb is what the test installs.
|
||||
workflows: ["Nightly GNU Build"]
|
||||
types: [completed]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# Dedicated pf-testing runner/environment: own concurrency group so perf runs
|
||||
# never block (or are blocked by) the pool-expansion / heal tests.
|
||||
concurrency:
|
||||
group: rustfs-performance-test
|
||||
cancel-in-progress: false
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
|
||||
env:
|
||||
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
|
||||
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
|
||||
# Performance test uses its own node list (4 nodes); the shared
|
||||
# RUSTFS_NODES secret is used by the 3-node pool-expansion / heal tests.
|
||||
RUSTFS_NODES: ${{ secrets.RUSTFS_PERF_NODES || vars.RUSTFS_PERF_NODES || 'vm000 vm001 vm002 vm003' }}
|
||||
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
|
||||
# Package used by the nightly run (workflow_dispatch inputs are empty for
|
||||
# workflow_run events), i.e. the latest nightly deb published by nightly-gnu.yml.
|
||||
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
|
||||
# Fixed benchmark result directory so later steps can read summary.md
|
||||
RUSTFS_RESULT_DIR: /tmp/rustfs-perf-results
|
||||
# Cross-repo token for writing to rustfs/backlog (set in repo settings)
|
||||
PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
jobs:
|
||||
performance-test:
|
||||
runs-on: pf-testing
|
||||
timeout-minutes: 900
|
||||
# Run on manual dispatch, or when the nightly build completed successfully.
|
||||
# Skipped when nightly failed.
|
||||
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
|
||||
steps:
|
||||
- name: Checkout auto-testing scripts
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: main
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
- name: Show environment
|
||||
run: |
|
||||
uname -a
|
||||
jq --version
|
||||
warp --version || true
|
||||
df -h /data | tail -1
|
||||
|
||||
- name: Reset test environment (before)
|
||||
if: ${{ inputs.cleanup_before != 'false' }}
|
||||
run: |
|
||||
chmod +x auto-testing/rustfs_performance_test.sh
|
||||
./auto-testing/rustfs_performance_test.sh --step 1 -y
|
||||
|
||||
- name: Install RustFS package & start cluster (4x4)
|
||||
run: |
|
||||
ARGS=(--steps "2,3,4" -y)
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_performance_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Preflight checks
|
||||
run: |
|
||||
ARGS=(--preflight)
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_performance_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Run benchmark (GET/PUT/MIXED)
|
||||
id: benchmark
|
||||
run: |
|
||||
# Empty on automatic (workflow_run) runs -> full 30 rounds.
|
||||
# Manual dispatch can restrict method(s)/size(s).
|
||||
export WARP_METHODS="${{ inputs.test_method }}"
|
||||
export WARP_SIZES="${{ inputs.object_size }}"
|
||||
./auto-testing/rustfs_performance_test.sh \
|
||||
--step 5 -y \
|
||||
--warp-duration "${{ inputs.warp_duration || '5m' }}" \
|
||||
--warp-concurrency "${{ inputs.warp_concurrency || '64' }}" \
|
||||
--log-file /tmp/rustfs-perf-test.log
|
||||
|
||||
- name: Analyze results
|
||||
if: ${{ steps.benchmark.conclusion == 'success' }}
|
||||
run: |
|
||||
./auto-testing/rustfs_performance_test.sh --step 6 -y
|
||||
|
||||
- name: Post results to backlog issue
|
||||
if: ${{ steps.benchmark.conclusion == 'success' }}
|
||||
env:
|
||||
GH_TOKEN: ${{ env.PF_TESTING_GH_TOKEN }}
|
||||
RESULT_DIR: ${{ env.RUSTFS_RESULT_DIR }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -z "${GH_TOKEN:-}" ]; then
|
||||
echo "PF_TESTING_GH_TOKEN is not configured; skipping issue post"
|
||||
exit 0
|
||||
fi
|
||||
SUMMARY="${RESULT_DIR}/summary.md"
|
||||
[ -f "${SUMMARY}" ] || { echo "summary.md not found at ${SUMMARY}"; exit 1; }
|
||||
DATE="$(date -u +%Y-%m-%d)"
|
||||
{
|
||||
echo "## RustFS nightly build performance testing report"
|
||||
echo ""
|
||||
echo "- **日期**: ${DATE}"
|
||||
echo "- **Run**: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
|
||||
echo "- **触发方式**: ${{ github.event_name }}"
|
||||
echo ""
|
||||
cat "${SUMMARY}"
|
||||
} > /tmp/rustfs-perf-issue-body.md
|
||||
TITLE="RustFS nightly build performance testing report"
|
||||
EXISTING="$(gh issue list --repo rustfs/backlog \
|
||||
--search "in:title \"${TITLE}\"" --state all --limit 5 \
|
||||
--json number --jq '.[0].number // empty')"
|
||||
if [ -n "${EXISTING}" ]; then
|
||||
gh issue comment "${EXISTING}" --repo rustfs/backlog --body-file /tmp/rustfs-perf-issue-body.md
|
||||
echo "commented on existing issue #${EXISTING}"
|
||||
else
|
||||
gh issue create --repo rustfs/backlog --title "${TITLE}" --body-file /tmp/rustfs-perf-issue-body.md
|
||||
fi
|
||||
|
||||
- name: Upload test logs & results
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
with:
|
||||
name: rustfs-perf-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-perf-test*.log
|
||||
/tmp/rustfs-perf-results/**
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Reset test environment (after)
|
||||
if: ${{ always() && inputs.cleanup_after != 'false' }}
|
||||
run: |
|
||||
./auto-testing/rustfs_performance_test.sh --step 7 -y
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: |
|
||||
echo "RustFS performance test failed"
|
||||
echo "Package source: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
|
||||
echo "See the uploaded log artifact for details."
|
||||
@@ -1,4 +1,4 @@
|
||||
name: RustFS Pool Expansion / Decommission Test
|
||||
name: RustFS Functional Test Suite (S3/KMS/tier/pool/heal)
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
@@ -30,6 +30,14 @@ on:
|
||||
description: 'Run the pool decommission step (3-pool topology only)'
|
||||
type: boolean
|
||||
default: true
|
||||
stop_node_gb:
|
||||
description: 'Heal: stop the outage node when surviving nodes reach N GiB'
|
||||
required: false
|
||||
default: '15'
|
||||
warp_stop_gb:
|
||||
description: 'Heal: stop warp when surviving nodes reach N GiB'
|
||||
required: false
|
||||
default: '40'
|
||||
cleanup_before:
|
||||
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
|
||||
type: boolean
|
||||
@@ -38,15 +46,17 @@ on:
|
||||
description: 'Reset the nodes after the test (DESTROYS test data/config)'
|
||||
type: boolean
|
||||
default: true
|
||||
schedule:
|
||||
# Nightly regression run; remove if you do not want a schedule.
|
||||
- cron: '0 21 * * *'
|
||||
workflow_run:
|
||||
# Run after the nightly build completes: S3 -> KMS -> tier -> pool -> heal.
|
||||
workflows: ["Nightly GNU Build"]
|
||||
types: [completed]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# Only one pool-expansion test at a time: the workflow mutates a shared
|
||||
# test environment, so concurrent runs must not clobber each other.
|
||||
# Only one test run at a time: every job mutates the same shared test
|
||||
# environment (vm000/vm001/vm002), so concurrent runs must not clobber each
|
||||
# other. Jobs inside a run are chained with needs to serialize them.
|
||||
concurrency:
|
||||
group: rustfs-pool-expansion-test
|
||||
cancel-in-progress: false
|
||||
@@ -61,19 +71,181 @@ env:
|
||||
RUSTFS_API_ENDPOINT: ${{ secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }}
|
||||
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
|
||||
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
|
||||
# Package used by the scheduled run (workflow_dispatch inputs are empty for
|
||||
# schedule events), i.e. the latest nightly deb published by nightly-gnu.yml.
|
||||
# Package used by the nightly run (workflow_dispatch inputs are empty for
|
||||
# workflow_run events), i.e. the latest nightly deb published by nightly-gnu.yml.
|
||||
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
|
||||
|
||||
jobs:
|
||||
pool-expansion-test:
|
||||
# All test scripts live in rustfs/auto-testing; rustfs stores only this
|
||||
# workflow. Each job checks out auto-testing before running.
|
||||
|
||||
s3-compat-test:
|
||||
name: S3 compatibility test
|
||||
runs-on: smoke-testing
|
||||
timeout-minutes: 360
|
||||
timeout-minutes: 240
|
||||
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
- name: Checkout auto-testing scripts
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: main
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
- name: Run S3 compatibility suite
|
||||
run: |
|
||||
chmod +x auto-testing/rustfs-s3-compat-test.sh
|
||||
ARGS=(--all-topologies -y --log-file /tmp/rustfs-s3-compat.log)
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
elif [ -n "${{ inputs.rustfs_version }}" ]; then
|
||||
ARGS+=(--version "${{ inputs.rustfs_version }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs-s3-compat-test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Upload test logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
with:
|
||||
name: rustfs-s3-compat-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-s3-compat*.log
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: |
|
||||
echo "RustFS S3 compatibility suite failed"
|
||||
echo "See the uploaded log artifact for details."
|
||||
|
||||
kms-test:
|
||||
name: KMS test (after S3)
|
||||
runs-on: smoke-testing
|
||||
timeout-minutes: 360
|
||||
needs: s3-compat-test
|
||||
if: ${{ always() && (github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success') }}
|
||||
steps:
|
||||
- name: Checkout auto-testing scripts
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: main
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
- name: Ensure docker (Vault container)
|
||||
run: |
|
||||
if ! command -v docker >/dev/null 2>&1; then
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y docker.io
|
||||
fi
|
||||
sudo systemctl enable --now docker
|
||||
docker info >/dev/null 2>&1 || sudo docker info >/dev/null 2>&1
|
||||
|
||||
- name: Run KMS suite
|
||||
run: |
|
||||
chmod +x auto-testing/rustfs-kms-test.sh
|
||||
ARGS=(--all-topologies --backends local,vault-kv2 -y --log-file /tmp/rustfs-kms.log)
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
elif [ -n "${{ inputs.rustfs_version }}" ]; then
|
||||
ARGS+=(--version "${{ inputs.rustfs_version }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs-kms-test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Upload test logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
with:
|
||||
name: rustfs-kms-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-kms*.log
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: |
|
||||
echo "RustFS KMS suite failed"
|
||||
echo "See the uploaded log artifact for details."
|
||||
|
||||
tier-test:
|
||||
name: Tier / event / audit test (after KMS)
|
||||
runs-on: smoke-testing
|
||||
timeout-minutes: 360
|
||||
needs: kms-test
|
||||
if: ${{ always() && (github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success') }}
|
||||
steps:
|
||||
- name: Checkout auto-testing scripts
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: main
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
- name: Ensure MQTT broker + clients (event notification tests)
|
||||
run: |
|
||||
if ! command -v mosquitto_sub >/dev/null 2>&1; then
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y mosquitto mosquitto-clients
|
||||
fi
|
||||
sudo mkdir -p /etc/mosquitto/conf.d
|
||||
printf 'listener 1883 0.0.0.0\nallow_anonymous true\n' | sudo tee /etc/mosquitto/conf.d/rustfs-test.conf >/dev/null
|
||||
sudo systemctl restart mosquitto
|
||||
sleep 2
|
||||
ss -tln 2>/dev/null | grep -q ':1883' || { echo "mosquitto not listening on 1883"; exit 1; }
|
||||
|
||||
- name: Run tier / event / audit suite
|
||||
run: |
|
||||
chmod +x auto-testing/rustfs-tier-test.sh
|
||||
ARGS=(--all-topologies -y --log-file /tmp/rustfs-tier.log)
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
elif [ -n "${{ inputs.rustfs_version }}" ]; then
|
||||
ARGS+=(--version "${{ inputs.rustfs_version }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs-tier-test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Upload test logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
with:
|
||||
name: rustfs-tier-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-tier*.log
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: |
|
||||
echo "RustFS tier/event/audit suite failed"
|
||||
echo "See the uploaded log artifact for details."
|
||||
|
||||
pool-expansion-test:
|
||||
name: Pool expansion / decommission test (after tier)
|
||||
runs-on: smoke-testing
|
||||
timeout-minutes: 360
|
||||
needs: tier-test
|
||||
if: ${{ always() && (github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success') }}
|
||||
steps:
|
||||
- name: Checkout auto-testing scripts
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: main
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
- name: Show environment
|
||||
run: |
|
||||
@@ -86,12 +258,12 @@ jobs:
|
||||
- name: Reset test environment (before)
|
||||
if: ${{ inputs.cleanup_before != 'false' }}
|
||||
run: |
|
||||
chmod +x scripts/test/rustfs_pool_expand.sh
|
||||
./scripts/test/rustfs_pool_expand.sh --reset -y
|
||||
chmod +x auto-testing/rustfs_pool_expand.sh
|
||||
./auto-testing/rustfs_pool_expand.sh --reset -y
|
||||
|
||||
- name: Install RustFS package & start first pool
|
||||
run: |
|
||||
ARGS=(--steps 1,2,3 -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
ARGS=(--steps "1,2,3" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
elif [ -n "${{ inputs.rustfs_version }}" ]; then
|
||||
@@ -99,7 +271,7 @@ jobs:
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./scripts/test/rustfs_pool_expand.sh "${ARGS[@]}"
|
||||
./auto-testing/rustfs_pool_expand.sh "${ARGS[@]}"
|
||||
|
||||
- name: Preflight checks
|
||||
run: |
|
||||
@@ -111,7 +283,7 @@ jobs:
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./scripts/test/rustfs_pool_expand.sh "${ARGS[@]}"
|
||||
./auto-testing/rustfs_pool_expand.sh "${ARGS[@]}"
|
||||
|
||||
- name: Run pool expansion & decommission test
|
||||
id: pool_test
|
||||
@@ -124,7 +296,7 @@ jobs:
|
||||
STEPS="$STEPS,9"
|
||||
fi
|
||||
fi
|
||||
./scripts/test/rustfs_pool_expand.sh \
|
||||
./auto-testing/rustfs_pool_expand.sh \
|
||||
--steps "$STEPS" --with-warp -y \
|
||||
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
|
||||
--storage-threshold "${{ inputs.storage_threshold || '50' }}" \
|
||||
@@ -137,14 +309,14 @@ jobs:
|
||||
with:
|
||||
name: rustfs-pool-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-pool-test.log
|
||||
/tmp/rustfs-warp.log
|
||||
/tmp/rustfs-pool-test*.log
|
||||
/tmp/rustfs-warp.*.log
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Reset test environment (after)
|
||||
if: ${{ always() && inputs.cleanup_after != 'false' }}
|
||||
run: |
|
||||
./scripts/test/rustfs_pool_expand.sh --reset -y
|
||||
./auto-testing/rustfs_pool_expand.sh --reset -y
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
@@ -152,3 +324,77 @@ jobs:
|
||||
echo "RustFS pool expansion test failed"
|
||||
echo "Package source: ${{ inputs.package_url || inputs.rustfs_version || 'nightly (R2 latest)' }}"
|
||||
echo "See the uploaded log artifact for details."
|
||||
|
||||
# Heal regression runs after the pool test regardless of its outcome: a pool
|
||||
# failure must be reported (it makes the run red) but must not block heal.
|
||||
heal-test:
|
||||
name: Heal test (after pool test)
|
||||
runs-on: smoke-testing
|
||||
timeout-minutes: 480
|
||||
needs: pool-expansion-test
|
||||
if: ${{ always() && (github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success') }}
|
||||
steps:
|
||||
- name: Checkout auto-testing scripts
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: main
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
- name: Reset test environment (before)
|
||||
if: ${{ inputs.cleanup_before != 'false' }}
|
||||
run: |
|
||||
chmod +x auto-testing/rustfs_heal_test.sh
|
||||
./auto-testing/rustfs_heal_test.sh --reset -y
|
||||
|
||||
- name: Install RustFS package & start cluster
|
||||
run: |
|
||||
ARGS=(--steps "1,2" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Preflight checks
|
||||
run: |
|
||||
ARGS=(--preflight --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Run heal test (write -> outage -> heal -> verify)
|
||||
run: |
|
||||
./auto-testing/rustfs_heal_test.sh \
|
||||
--steps 3,4,5,6,7 -y \
|
||||
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
|
||||
--stop-node-gb "${{ inputs.stop_node_gb || '15' }}" \
|
||||
--warp-stop-gb "${{ inputs.warp_stop_gb || '40' }}" \
|
||||
--log-file /tmp/rustfs-heal-test.log
|
||||
|
||||
- name: Upload test logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
with:
|
||||
name: rustfs-heal-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-heal-test.log
|
||||
/tmp/rustfs-warp.*.log
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Reset test environment (after)
|
||||
if: ${{ always() && inputs.cleanup_after != 'false' }}
|
||||
run: |
|
||||
./auto-testing/rustfs_heal_test.sh --reset -y
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: |
|
||||
echo "RustFS heal test failed"
|
||||
echo "See the uploaded log artifact for details."
|
||||
|
||||
@@ -31,8 +31,13 @@ This file contains repository-wide rules. Use the nearest subdirectory
|
||||
- An existing clean, isolated task worktree is sufficient. Create another
|
||||
worktree only when the current checkout is shared, dirty with unrelated work,
|
||||
or belongs to another task.
|
||||
- Never commit from a shared checkout. Use an `overtrue/` feature branch unless
|
||||
the user requests another name.
|
||||
- Never commit from a shared checkout.
|
||||
- Use a task-specific branch named `<type>/<topic>`, such as `fix/...`,
|
||||
`feat/...`, `test/...`, or `docs/...`, unless the user specifies a name.
|
||||
- Do not include agent, tool, contributor, account, or organization names in
|
||||
branch names.
|
||||
- Push to the user-requested remote or the repository's configured push remote.
|
||||
Do not hard-code or infer a remote from an account name.
|
||||
- Check free space before artifact-heavy builds, tests, coverage, or downloads.
|
||||
Re-check before a broad gate when space is tight.
|
||||
- Remove only task-owned temporary/build artifacts. Never delete another task's
|
||||
|
||||
Generated
+62
-44
@@ -62,7 +62,7 @@ checksum = "f8eb277bec05f56a0e0591f155a484cbd0f4f07ff2905051a48c72f004f7ed58"
|
||||
dependencies = [
|
||||
"cipher 0.5.2",
|
||||
"cpubits",
|
||||
"cpufeatures 0.3.0",
|
||||
"cpufeatures 0.3.1",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
@@ -323,13 +323,13 @@ checksum = "03918c3dbd7701a85c6b9887732e2921175f26c350b4563841d0958c21d57e6d"
|
||||
|
||||
[[package]]
|
||||
name = "argon2"
|
||||
version = "0.6.0-rc.8"
|
||||
version = "0.6.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7af50940b73bf4e16c15c448a2b121c63f2d68e3e54b6a8731673cb4aa0cdff5"
|
||||
checksum = "134c52ddac6d63c576bef8168db10c83c49c26444ecbc68060fef078925a901c"
|
||||
dependencies = [
|
||||
"base64ct",
|
||||
"blake2",
|
||||
"cpufeatures 0.3.0",
|
||||
"cpufeatures 0.3.1",
|
||||
"password-hash",
|
||||
]
|
||||
|
||||
@@ -1024,7 +1024,7 @@ dependencies = [
|
||||
"http 0.2.12",
|
||||
"http 1.5.0",
|
||||
"http-body 1.1.0",
|
||||
"lru 0.18.2",
|
||||
"lru 0.18.3",
|
||||
"percent-encoding",
|
||||
"regex-lite",
|
||||
"sha2 0.11.0",
|
||||
@@ -1939,13 +1939,13 @@ checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527"
|
||||
|
||||
[[package]]
|
||||
name = "chacha20"
|
||||
version = "0.10.1"
|
||||
version = "0.10.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81"
|
||||
checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cipher 0.5.2",
|
||||
"cpufeatures 0.3.0",
|
||||
"cpufeatures 0.3.1",
|
||||
"rand_core 0.10.1",
|
||||
"zeroize",
|
||||
]
|
||||
@@ -2283,9 +2283,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "convert_case"
|
||||
version = "0.11.0"
|
||||
version = "0.12.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "affbf0190ed2caf063e3def54ff444b449371d55c58e513a95ab98eca50adb49"
|
||||
checksum = "1af709f1f33454bf52eadfc8c78b3b9ef9cb26fb54d16dc9cd9a7299f899fd1b"
|
||||
dependencies = [
|
||||
"unicode-segmentation",
|
||||
]
|
||||
@@ -2348,9 +2348,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "cpufeatures"
|
||||
version = "0.3.0"
|
||||
version = "0.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201"
|
||||
checksum = "5ca28b0ae3115b884660db4118d803791fd6756b6e88f39c0f3f7859060d7566"
|
||||
dependencies = [
|
||||
"libc",
|
||||
]
|
||||
@@ -2611,7 +2611,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b5eed333089e2e1c1ac8c6c0398e5e2497b4c9926ca6d0365ed1e099afa5bc23"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures 0.3.0",
|
||||
"cpufeatures 0.3.1",
|
||||
"curve25519-dalek-derive",
|
||||
"digest 0.11.3",
|
||||
"fiat-crypto 0.3.0",
|
||||
@@ -5885,7 +5885,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d8f198d1db720e4940b5a493201d199d9f24f568f8f746bd13706243a2f71598"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures 0.3.0",
|
||||
"cpufeatures 0.3.1",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -6103,9 +6103,9 @@ checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981"
|
||||
|
||||
[[package]]
|
||||
name = "libredox"
|
||||
version = "0.1.20"
|
||||
version = "0.1.21"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "28d0a00925a9f930d679b6789b721e3a7f9ed110f41b86d2497caa780c3a070a"
|
||||
checksum = "d7955dfc218a8afb29dfeffd540e3a6e96baeb94fe7138228dd7cc6937fbbf96"
|
||||
dependencies = [
|
||||
"libc",
|
||||
]
|
||||
@@ -6209,9 +6209,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "lru"
|
||||
version = "0.18.2"
|
||||
version = "0.18.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5d2f2f9b4ba7e6b24d95e7e899329d35be83bcded72c8540cdd5368932d1d90a"
|
||||
checksum = "0d317b4b9eb398e6acce275758ec6125535505e7a146fb1a9b8bda2451b0ff4c"
|
||||
dependencies = [
|
||||
"hashbrown 0.17.1",
|
||||
]
|
||||
@@ -6659,7 +6659,7 @@ dependencies = [
|
||||
"futures-sink",
|
||||
"futures-util",
|
||||
"keyed_priority_queue",
|
||||
"lru 0.18.2",
|
||||
"lru 0.18.3",
|
||||
"mysql_common",
|
||||
"percent-encoding",
|
||||
"rand 0.10.2",
|
||||
@@ -7371,9 +7371,9 @@ checksum = "1a80800c0488c3a21695ea981a54918fbb37abf04f4d0720c453632255e2ff0e"
|
||||
|
||||
[[package]]
|
||||
name = "owo-colors"
|
||||
version = "4.3.0"
|
||||
version = "4.4.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d211803b9b6b570f68772237e415a029d5a50c65d382910b879fb19d3271f94d"
|
||||
checksum = "13c45bb4a6ae1280ec0803b1ef9d3455eb50f01efbbe1447ab020f1d54fba9d8"
|
||||
|
||||
[[package]]
|
||||
name = "p12-keystore"
|
||||
@@ -7924,7 +7924,7 @@ version = "0.9.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6e2d0073b297041425c7c3df6eb4792d598a15323fe63346852b092eca02904c"
|
||||
dependencies = [
|
||||
"cpufeatures 0.3.0",
|
||||
"cpufeatures 0.3.1",
|
||||
"universal-hash",
|
||||
"zeroize",
|
||||
]
|
||||
@@ -7936,7 +7936,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f0fa31d631f2b2cb2a544d0aa321ce847a94764d701ca2becc411138b93d49cd"
|
||||
dependencies = [
|
||||
"cpubits",
|
||||
"cpufeatures 0.3.0",
|
||||
"cpufeatures 0.3.1",
|
||||
"universal-hash",
|
||||
"zeroize",
|
||||
]
|
||||
@@ -8512,16 +8512,6 @@ version = "1.2.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0"
|
||||
|
||||
[[package]]
|
||||
name = "quick-xml"
|
||||
version = "0.41.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e660451e55124f798a69a5af3f49ccfbefbd41910eefd25caf2393e1f3473ec1"
|
||||
dependencies = [
|
||||
"memchr",
|
||||
"serde",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quick-xml"
|
||||
version = "0.42.0"
|
||||
@@ -9414,7 +9404,7 @@ dependencies = [
|
||||
"percent-encoding",
|
||||
"pin-project-lite",
|
||||
"proptest",
|
||||
"quick-xml 0.42.0",
|
||||
"quick-xml",
|
||||
"rand 0.10.2",
|
||||
"rcgen",
|
||||
"regex",
|
||||
@@ -10157,7 +10147,7 @@ dependencies = [
|
||||
"jiff",
|
||||
"metrics",
|
||||
"percent-encoding",
|
||||
"quick-xml 0.42.0",
|
||||
"quick-xml",
|
||||
"rayon",
|
||||
"rustc-hash",
|
||||
"rustfs-config",
|
||||
@@ -10489,7 +10479,7 @@ dependencies = [
|
||||
"hyper-util",
|
||||
"lazy_static",
|
||||
"md-5 0.11.0",
|
||||
"quick-xml 0.42.0",
|
||||
"quick-xml",
|
||||
"rand 0.10.2",
|
||||
"rustfs-checksums",
|
||||
"rustfs-config",
|
||||
@@ -10611,6 +10601,7 @@ dependencies = [
|
||||
"s3s",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"serial_test",
|
||||
"sha2 0.11.0",
|
||||
"temp-env",
|
||||
"tempfile",
|
||||
@@ -10815,7 +10806,7 @@ dependencies = [
|
||||
"blake2",
|
||||
"brotli",
|
||||
"bytes",
|
||||
"convert_case 0.11.0",
|
||||
"convert_case 0.12.0",
|
||||
"crc-fast",
|
||||
"criterion",
|
||||
"flate2",
|
||||
@@ -11042,7 +11033,7 @@ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
|
||||
[[package]]
|
||||
name = "s3s"
|
||||
version = "0.15.0"
|
||||
source = "git+https://github.com/rustfs/s3s.git?rev=0f6f83d98b37fd9edcaa3be573db4aa8f568e088#0f6f83d98b37fd9edcaa3be573db4aa8f568e088"
|
||||
source = "git+https://github.com/rustfs/s3s.git?rev=6e7b41252c7ba218a90886f58d297716ddf68acf#6e7b41252c7ba218a90886f58d297716ddf68acf"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"arrayvec",
|
||||
@@ -11063,14 +11054,17 @@ dependencies = [
|
||||
"httparse",
|
||||
"hyper",
|
||||
"itoa",
|
||||
"jiff",
|
||||
"md-5 0.11.0",
|
||||
"memchr",
|
||||
"mime",
|
||||
"nom 8.0.0",
|
||||
"numeric_cast",
|
||||
"pin-project-lite",
|
||||
"quick-xml 0.41.0",
|
||||
"quick-xml",
|
||||
"regex",
|
||||
"s3s-sigv2",
|
||||
"s3s-sigv4",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"serde_urlencoded",
|
||||
@@ -11092,6 +11086,30 @@ dependencies = [
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "s3s-sigv2"
|
||||
version = "0.16.0-alpha.1"
|
||||
source = "git+https://github.com/rustfs/s3s.git?rev=6e7b41252c7ba218a90886f58d297716ddf68acf#6e7b41252c7ba218a90886f58d297716ddf68acf"
|
||||
dependencies = [
|
||||
"jiff",
|
||||
"thiserror 2.0.20",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "s3s-sigv4"
|
||||
version = "0.16.0-alpha.1"
|
||||
source = "git+https://github.com/rustfs/s3s.git?rev=6e7b41252c7ba218a90886f58d297716ddf68acf#6e7b41252c7ba218a90886f58d297716ddf68acf"
|
||||
dependencies = [
|
||||
"arrayvec",
|
||||
"base64-simd",
|
||||
"hex-simd",
|
||||
"jiff",
|
||||
"nom 8.0.0",
|
||||
"serde",
|
||||
"smallvec",
|
||||
"thiserror 2.0.20",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "salsa20"
|
||||
version = "0.10.2"
|
||||
@@ -11473,7 +11491,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures 0.3.0",
|
||||
"cpufeatures 0.3.1",
|
||||
"digest 0.11.3",
|
||||
]
|
||||
|
||||
@@ -11501,7 +11519,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures 0.3.0",
|
||||
"cpufeatures 0.3.1",
|
||||
"digest 0.11.3",
|
||||
]
|
||||
|
||||
@@ -12174,7 +12192,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
|
||||
dependencies = [
|
||||
"fastrand",
|
||||
"getrandom 0.4.3",
|
||||
"getrandom 0.3.4",
|
||||
"once_cell",
|
||||
"rustix",
|
||||
"windows-sys 0.61.2",
|
||||
@@ -12864,9 +12882,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "twox-hash"
|
||||
version = "2.1.3"
|
||||
version = "2.1.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8464ec13c3691491391d9fce00f6416c9a48e46972f72d7865688be2080192c9"
|
||||
checksum = "5283634e518fe9e82c7b20520bb4bc209009fd16c82077c802f8111ecbb0117a"
|
||||
|
||||
[[package]]
|
||||
name = "typed-path"
|
||||
|
||||
+3
-3
@@ -198,7 +198,7 @@ serde_urlencoded = "0.7.1"
|
||||
# have incompatible APIs. Keep them exact-pinned and monitor upstream for stable
|
||||
# releases.
|
||||
aes-gcm = { version = "=0.11.1" }
|
||||
argon2 = { version = "=0.6.0-rc.8" }
|
||||
argon2 = { version = "=0.6.0" }
|
||||
blake2 = "=0.11.0"
|
||||
chacha20poly1305 = { version = "=0.11.0" }
|
||||
crc-fast = "1.10.0"
|
||||
@@ -247,7 +247,7 @@ base64-simd = "0.8.0"
|
||||
brotli = "8.0.4"
|
||||
clap = { version = "4.6.6" }
|
||||
const-str = { version = "1.1.0" }
|
||||
convert_case = "0.11.0"
|
||||
convert_case = "0.12.0"
|
||||
criterion = { version = "0.8" }
|
||||
crossbeam-queue = "0.3.13"
|
||||
crossbeam-channel = "0.5.16"
|
||||
@@ -304,7 +304,7 @@ rustify = { version = "0.7", default-features = false }
|
||||
rustix = { version = "1.1.4" }
|
||||
rust-embed = { version = "8.12.0" }
|
||||
rustc-hash = { version = "2.1.3" }
|
||||
s3s = { git = "https://github.com/rustfs/s3s.git", rev = "0f6f83d98b37fd9edcaa3be573db4aa8f568e088", version = "0.15.0", features = ["minio"] }
|
||||
s3s = { git = "https://github.com/rustfs/s3s.git", rev = "6e7b41252c7ba218a90886f58d297716ddf68acf", version = "0.15.0", features = ["minio"] }
|
||||
serial_test = "4.0.1"
|
||||
shadow-rs = { default-features = false, version = "2.0.0" }
|
||||
siphasher = "1.0.3"
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://docs.rustfs.com/installation/">Getting Started</a>
|
||||
<a href="https://docs.rustfs.com/en/installation">Getting Started</a>
|
||||
· <a href="https://docs.rustfs.com/">Docs</a>
|
||||
· <a href="https://github.com/rustfs/rustfs/issues">Bug reports</a>
|
||||
· <a href="https://github.com/rustfs/rustfs/discussions">Discussions</a>
|
||||
|
||||
+1
-1
@@ -16,7 +16,7 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://docs.rustfs.com/installation/">快速开始</a>
|
||||
<a href="https://docs.rustfs.com/zh/installation">快速开始</a>
|
||||
· <a href="https://docs.rustfs.com/">文档</a>
|
||||
· <a href="https://github.com/rustfs/rustfs/issues">报告 Bug</a>
|
||||
· <a href="https://github.com/rustfs/rustfs/discussions">社区讨论</a>
|
||||
|
||||
@@ -178,6 +178,76 @@ pub trait WorkloadAdmissionSnapshotProvider {
|
||||
fn workload_admission_snapshot(&self) -> WorkloadAdmissionRegistrySnapshot;
|
||||
}
|
||||
|
||||
/// Foreground workload pressure observed against a configured utilization threshold.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct ForegroundPressure {
|
||||
/// Foreground workload class whose utilization reached its threshold.
|
||||
pub class: WorkloadClass,
|
||||
/// Observed utilization percentage for the class.
|
||||
pub usage_pct: usize,
|
||||
/// Configured threshold percentage that the observed utilization reached.
|
||||
pub threshold_pct: usize,
|
||||
}
|
||||
|
||||
impl ForegroundPressure {
|
||||
/// Return a stable reason label for logs and metrics.
|
||||
pub const fn reason(self) -> &'static str {
|
||||
match self.class {
|
||||
WorkloadClass::ForegroundRead => "foreground_read_pressure",
|
||||
WorkloadClass::ForegroundWrite => "foreground_write_pressure",
|
||||
_ => "foreground_pressure",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Return the strongest foreground pressure in `snapshot`, if any.
|
||||
///
|
||||
/// A zero threshold disables its class. `Saturated` counts as full utilization
|
||||
/// regardless of the reported limit; otherwise a class contributes only when it
|
||||
/// reports a non-zero limit, with a missing active count read as zero. When both
|
||||
/// classes are above their threshold the higher utilization wins.
|
||||
///
|
||||
/// Callers own the enable switch: this function evaluates thresholds only.
|
||||
pub fn foreground_pressure(
|
||||
snapshot: &WorkloadAdmissionRegistrySnapshot,
|
||||
read_threshold_pct: usize,
|
||||
write_threshold_pct: usize,
|
||||
) -> Option<ForegroundPressure> {
|
||||
[
|
||||
(WorkloadClass::ForegroundRead, read_threshold_pct),
|
||||
(WorkloadClass::ForegroundWrite, write_threshold_pct),
|
||||
]
|
||||
.into_iter()
|
||||
.filter_map(|(class, threshold_pct)| {
|
||||
if threshold_pct == 0 {
|
||||
return None;
|
||||
}
|
||||
|
||||
let entry = snapshot.get(class)?;
|
||||
let usage_pct = if matches!(entry.state, AdmissionState::Saturated) {
|
||||
100
|
||||
} else {
|
||||
let limit = entry.limit?;
|
||||
if limit == 0 {
|
||||
return None;
|
||||
}
|
||||
entry
|
||||
.active
|
||||
.unwrap_or(0)
|
||||
.saturating_mul(100)
|
||||
.checked_div(limit)
|
||||
.unwrap_or(100)
|
||||
};
|
||||
|
||||
(usage_pct >= threshold_pct).then_some(ForegroundPressure {
|
||||
class,
|
||||
usage_pct,
|
||||
threshold_pct,
|
||||
})
|
||||
})
|
||||
.max_by_key(|pressure| pressure.usage_pct)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
@@ -314,4 +384,205 @@ mod tests {
|
||||
|
||||
assert!(err.to_string().contains("unexpected"));
|
||||
}
|
||||
|
||||
fn counted(
|
||||
class: WorkloadClass,
|
||||
state: AdmissionState,
|
||||
active: Option<usize>,
|
||||
limit: Option<usize>,
|
||||
) -> WorkloadAdmissionSnapshot {
|
||||
WorkloadAdmissionSnapshot::new(class, state).with_counts(active, None, limit)
|
||||
}
|
||||
|
||||
fn registry(entries: Vec<WorkloadAdmissionSnapshot>) -> WorkloadAdmissionRegistrySnapshot {
|
||||
WorkloadAdmissionRegistrySnapshot::new(entries)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_reason_labels_cover_non_foreground_classes() {
|
||||
let read = ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundRead,
|
||||
usage_pct: 90,
|
||||
threshold_pct: 80,
|
||||
};
|
||||
let write = ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundWrite,
|
||||
usage_pct: 90,
|
||||
threshold_pct: 80,
|
||||
};
|
||||
let repair = ForegroundPressure {
|
||||
class: WorkloadClass::Repair,
|
||||
usage_pct: 90,
|
||||
threshold_pct: 80,
|
||||
};
|
||||
|
||||
assert_eq!(read.reason(), "foreground_read_pressure");
|
||||
assert_eq!(write.reason(), "foreground_write_pressure");
|
||||
assert_eq!(repair.reason(), "foreground_pressure");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_is_disabled_when_both_thresholds_are_zero() {
|
||||
let snapshot = registry(vec![
|
||||
counted(WorkloadClass::ForegroundRead, AdmissionState::Saturated, Some(8), Some(8)),
|
||||
counted(WorkloadClass::ForegroundWrite, AdmissionState::Saturated, Some(8), Some(8)),
|
||||
]);
|
||||
|
||||
assert_eq!(foreground_pressure(&snapshot, 0, 0), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_skips_only_the_class_whose_threshold_is_zero() {
|
||||
let snapshot = registry(vec![
|
||||
counted(WorkloadClass::ForegroundRead, AdmissionState::Open, Some(10), Some(10)),
|
||||
counted(WorkloadClass::ForegroundWrite, AdmissionState::Open, Some(9), Some(10)),
|
||||
]);
|
||||
|
||||
assert_eq!(
|
||||
foreground_pressure(&snapshot, 0, 80),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundWrite,
|
||||
usage_pct: 90,
|
||||
threshold_pct: 80,
|
||||
})
|
||||
);
|
||||
assert_eq!(
|
||||
foreground_pressure(&snapshot, 80, 0),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundRead,
|
||||
usage_pct: 100,
|
||||
threshold_pct: 80,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_ignores_missing_entries() {
|
||||
let snapshot = registry(vec![counted(WorkloadClass::Scanner, AdmissionState::Saturated, Some(8), Some(8))]);
|
||||
|
||||
assert_eq!(foreground_pressure(&snapshot, 1, 1), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_ignores_missing_and_zero_limits() {
|
||||
let missing_limit = registry(vec![counted(
|
||||
WorkloadClass::ForegroundRead,
|
||||
AdmissionState::Throttled,
|
||||
Some(8),
|
||||
None,
|
||||
)]);
|
||||
let zero_limit = registry(vec![counted(
|
||||
WorkloadClass::ForegroundWrite,
|
||||
AdmissionState::Throttled,
|
||||
Some(8),
|
||||
Some(0),
|
||||
)]);
|
||||
|
||||
assert_eq!(foreground_pressure(&missing_limit, 1, 1), None);
|
||||
assert_eq!(foreground_pressure(&zero_limit, 1, 1), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_treats_saturated_as_full_without_reading_limit() {
|
||||
let snapshot = registry(vec![
|
||||
counted(WorkloadClass::ForegroundRead, AdmissionState::Saturated, None, None),
|
||||
counted(WorkloadClass::ForegroundWrite, AdmissionState::Saturated, Some(0), Some(0)),
|
||||
]);
|
||||
|
||||
assert_eq!(
|
||||
foreground_pressure(&snapshot, 100, 0),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundRead,
|
||||
usage_pct: 100,
|
||||
threshold_pct: 100,
|
||||
})
|
||||
);
|
||||
assert_eq!(
|
||||
foreground_pressure(&snapshot, 0, 100),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundWrite,
|
||||
usage_pct: 100,
|
||||
threshold_pct: 100,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_reads_missing_active_as_zero() {
|
||||
let snapshot = registry(vec![counted(WorkloadClass::ForegroundRead, AdmissionState::Open, None, Some(8))]);
|
||||
|
||||
assert_eq!(foreground_pressure(&snapshot, 1, 1), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_returns_the_higher_utilization_when_both_classes_exceed() {
|
||||
let read_higher = registry(vec![
|
||||
counted(WorkloadClass::ForegroundRead, AdmissionState::Open, Some(19), Some(20)),
|
||||
counted(WorkloadClass::ForegroundWrite, AdmissionState::Open, Some(17), Some(20)),
|
||||
]);
|
||||
let write_higher = registry(vec![
|
||||
counted(WorkloadClass::ForegroundRead, AdmissionState::Open, Some(17), Some(20)),
|
||||
counted(WorkloadClass::ForegroundWrite, AdmissionState::Open, Some(19), Some(20)),
|
||||
]);
|
||||
|
||||
assert_eq!(
|
||||
foreground_pressure(&read_higher, 80, 80),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundRead,
|
||||
usage_pct: 95,
|
||||
threshold_pct: 80,
|
||||
})
|
||||
);
|
||||
assert_eq!(
|
||||
foreground_pressure(&write_higher, 80, 80),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundWrite,
|
||||
usage_pct: 95,
|
||||
threshold_pct: 80,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_breaks_utilization_ties_toward_the_write_class() {
|
||||
let snapshot = registry(vec![
|
||||
counted(WorkloadClass::ForegroundRead, AdmissionState::Open, Some(18), Some(20)),
|
||||
counted(WorkloadClass::ForegroundWrite, AdmissionState::Open, Some(18), Some(20)),
|
||||
]);
|
||||
|
||||
assert_eq!(
|
||||
foreground_pressure(&snapshot, 80, 80),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundWrite,
|
||||
usage_pct: 90,
|
||||
threshold_pct: 80,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_triggers_exactly_at_the_threshold_and_not_below() {
|
||||
let at_threshold = registry(vec![counted(
|
||||
WorkloadClass::ForegroundRead,
|
||||
AdmissionState::Open,
|
||||
Some(8),
|
||||
Some(10),
|
||||
)]);
|
||||
let below_threshold = registry(vec![counted(
|
||||
WorkloadClass::ForegroundRead,
|
||||
AdmissionState::Open,
|
||||
Some(7),
|
||||
Some(10),
|
||||
)]);
|
||||
|
||||
assert_eq!(
|
||||
foreground_pressure(&at_threshold, 80, 80),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundRead,
|
||||
usage_pct: 80,
|
||||
threshold_pct: 80,
|
||||
})
|
||||
);
|
||||
assert_eq!(foreground_pressure(&below_threshold, 80, 80), None);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -40,6 +40,14 @@ pub const HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS: u64 = 5_000;
|
||||
pub const ENV_HEALTH_CLUSTER_TIMEOUT_MS: &str = "RUSTFS_HEALTH_CLUSTER_TIMEOUT_MS";
|
||||
pub const DEFAULT_HEALTH_CLUSTER_TIMEOUT_MS: u64 = 2000;
|
||||
|
||||
/// Timeout for one remote lock-client online check used by readiness (milliseconds).
|
||||
///
|
||||
/// This is intentionally shorter than the generic lock RPC timeout so
|
||||
/// `/health/ready` can report degradation instead of riding a dead peer's
|
||||
/// connect or HTTP/2 keepalive budget.
|
||||
pub const ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS: &str = "RUSTFS_HEALTH_LOCK_ONLINE_TIMEOUT_MS";
|
||||
pub const DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS: u64 = 1000;
|
||||
|
||||
/// Maximum time to wait for local node runtime readiness (storage / IAM / lock
|
||||
/// quorum) during startup before failing fast (seconds).
|
||||
///
|
||||
|
||||
@@ -288,6 +288,49 @@ pub const DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 0;
|
||||
|
||||
const _: () = assert!(!DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE);
|
||||
|
||||
/// Enable automatic foreground admission for large or unknown-size PutObject requests.
|
||||
///
|
||||
/// Unlike the strict experimental gate above, this default-on path only applies
|
||||
/// to requests that are large enough to create sustained erasure/RPC pressure.
|
||||
/// Small PUTs continue on the legacy path unless the strict gate is explicitly
|
||||
/// enabled.
|
||||
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE";
|
||||
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: bool = true;
|
||||
|
||||
/// Maximum automatic foreground write requests admitted concurrently per process.
|
||||
///
|
||||
/// `0` derives a conservative default from the local disk-read scheduler cap,
|
||||
/// currently clamped to protect the commit path without making ordinary high
|
||||
/// throughput uploads single-file.
|
||||
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT";
|
||||
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: usize = 0;
|
||||
|
||||
/// Minimum direct PutObject size that enters automatic foreground write admission.
|
||||
///
|
||||
/// Requests with an unknown size are treated as large because the write pressure
|
||||
/// cannot be bounded from headers.
|
||||
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES";
|
||||
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 32 * 1024 * 1024;
|
||||
|
||||
/// Minimum UploadPart size that enters automatic foreground write admission.
|
||||
///
|
||||
/// Multipart pressure is often many moderate-sized parts rather than one very
|
||||
/// large request. The default gates every multipart part through the same permit
|
||||
/// pool as large/unknown-size PutObject while keeping small direct PUTs on the
|
||||
/// legacy path.
|
||||
pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: &str =
|
||||
"RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES";
|
||||
pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 0;
|
||||
|
||||
/// Time in milliseconds an automatic foreground write waits for a permit.
|
||||
///
|
||||
/// A short wait smooths transient bursts while still returning S3
|
||||
/// `SlowDown`/503 before body ingest when the node is already saturated.
|
||||
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
|
||||
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 250;
|
||||
|
||||
const _: () = assert!(DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE);
|
||||
|
||||
/// Environment variable for minimum GetObject timeout in seconds.
|
||||
///
|
||||
/// When dynamic timeout calculation is enabled, this is the minimum timeout
|
||||
|
||||
@@ -31,14 +31,9 @@
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client, rustfs_binary_path};
|
||||
use aws_sdk_s3::config::{Credentials, Region};
|
||||
use crate::common::{RustFSTestEnvironment, init_logging, rustfs_binary_path};
|
||||
use aws_sdk_s3::Client;
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
use aws_sdk_s3::{Client, Config};
|
||||
use http::header::HOST;
|
||||
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
|
||||
use rustfs_signer::sign_v4;
|
||||
use s3s::Body;
|
||||
use std::error::Error;
|
||||
use std::io::Read;
|
||||
use std::process::{Command, Stdio};
|
||||
@@ -87,10 +82,10 @@ mod tests {
|
||||
}
|
||||
|
||||
/// Send a SigV4-signed request to `path` (optionally with a JSON `body`) and
|
||||
/// return `(status, body)`. Uses the `UNSIGNED_PAYLOAD` content hash so a
|
||||
/// request body can be attached without the caller pre-hashing it — the
|
||||
/// server verifies the signature against the same sentinel, exactly as the
|
||||
/// AWS SDKs / MinIO client do for streaming/unsigned payloads.
|
||||
/// return `(status, body)`.
|
||||
///
|
||||
/// Thin wrapper over [`crate::common::admin_request`], kept local so the
|
||||
/// call sites below keep their `Option<&str>` body shape.
|
||||
async fn signed_request(
|
||||
base_url: &str,
|
||||
method: http::Method,
|
||||
@@ -99,47 +94,13 @@ mod tests {
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
) -> Result<(reqwest::StatusCode, String), Box<dyn Error + Send + Sync>> {
|
||||
let url = format!("{base_url}{path}");
|
||||
let uri = url.parse::<http::Uri>()?;
|
||||
let authority = uri.authority().ok_or("missing authority")?.to_string();
|
||||
let body_bytes = body.map(|b| b.as_bytes().to_vec()).unwrap_or_default();
|
||||
|
||||
// The signature is computed over `UNSIGNED_PAYLOAD`, so the body bytes do
|
||||
// not participate in the SigV4 hash — sign over an empty body and attach
|
||||
// the real payload to the wire request below.
|
||||
let request = http::Request::builder()
|
||||
.method(method.clone())
|
||||
.uri(uri)
|
||||
.header(HOST, authority)
|
||||
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
|
||||
let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
|
||||
|
||||
let client = local_http_client();
|
||||
let mut rb = client.request(method, url.as_str());
|
||||
for (name, value) in signed.headers() {
|
||||
rb = rb.header(name, value);
|
||||
}
|
||||
if !body_bytes.is_empty() {
|
||||
rb = rb.body(body_bytes);
|
||||
}
|
||||
let resp = rb.send().await?;
|
||||
let status = resp.status();
|
||||
let text = resp.text().await?;
|
||||
Ok((status, text))
|
||||
crate::common::admin_request(base_url, method, path, body.map(str::to_string), access_key, secret_key).await
|
||||
}
|
||||
|
||||
/// Build an S3 client bound to explicit credentials (used to exercise the S3
|
||||
/// data plane with rotated / stale root credentials).
|
||||
fn s3_client_with(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
|
||||
let credentials = Credentials::new(access_key, secret_key, None, None, "sec4-admin-auth");
|
||||
let config = Config::builder()
|
||||
.credentials_provider(credentials)
|
||||
.region(Region::new("us-east-1"))
|
||||
.endpoint_url(&env.url)
|
||||
.force_path_style(true)
|
||||
.behavior_version_latest()
|
||||
.build();
|
||||
Client::from_conf(config)
|
||||
env.create_s3_client_with_credentials(access_key, secret_key)
|
||||
}
|
||||
|
||||
/// Create a non-admin IAM user via the admin `add-user` API using the root
|
||||
@@ -151,12 +112,7 @@ mod tests {
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
let path = format!("/rustfs/admin/v3/add-user?accessKey={access_key}");
|
||||
let body = serde_json::json!({ "secretKey": secret_key, "status": "enabled" }).to_string();
|
||||
let (status, resp) =
|
||||
signed_request(&env.url, http::Method::PUT, &path, Some(&body), &env.access_key, &env.secret_key).await?;
|
||||
assert!(status.is_success(), "add-user should succeed (status={status}, body={resp})");
|
||||
Ok(())
|
||||
crate::common::admin_create_user(env, access_key, secret_key).await
|
||||
}
|
||||
|
||||
/// A fully authenticated but non-admin credential must be rejected with
|
||||
|
||||
@@ -59,8 +59,8 @@ mod tests {
|
||||
|
||||
/// One signed admin request, returning the status and the raw body.
|
||||
///
|
||||
/// Signs with `UNSIGNED_PAYLOAD` so the body does not participate in the
|
||||
/// hash, matching how the other admin e2e tests drive these routes.
|
||||
/// Thin wrapper over [`crate::common::admin_request`], kept local so the
|
||||
/// call sites below keep their `Option<&str>` body shape.
|
||||
async fn signed_request(
|
||||
base_url: &str,
|
||||
method: http::Method,
|
||||
@@ -69,30 +69,7 @@ mod tests {
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
) -> Result<(reqwest::StatusCode, String), Box<dyn Error + Send + Sync>> {
|
||||
let url = format!("{base_url}{path}");
|
||||
let uri = url.parse::<http::Uri>()?;
|
||||
let authority = uri.authority().ok_or("missing authority")?.to_string();
|
||||
let body_bytes = body.map(|b| b.as_bytes().to_vec()).unwrap_or_default();
|
||||
|
||||
let request = http::Request::builder()
|
||||
.method(method.clone())
|
||||
.uri(uri)
|
||||
.header(HOST, authority)
|
||||
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
|
||||
let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
|
||||
|
||||
let client = local_http_client();
|
||||
let mut builder = client.request(method, url.as_str());
|
||||
for (name, value) in signed.headers() {
|
||||
builder = builder.header(name, value);
|
||||
}
|
||||
if !body_bytes.is_empty() {
|
||||
builder = builder.body(body_bytes);
|
||||
}
|
||||
let response = builder.send().await?;
|
||||
let status = response.status();
|
||||
let text = response.text().await?;
|
||||
Ok((status, text))
|
||||
crate::common::admin_request(base_url, method, path, body.map(str::to_string), access_key, secret_key).await
|
||||
}
|
||||
|
||||
/// A SigV4-signed `AssumeRole` form POST, optionally carrying a second factor.
|
||||
|
||||
@@ -15,39 +15,23 @@
|
||||
//! Regression test for Issue #1423
|
||||
//! Verifies that Bucket Policies are honored for Authenticated Users.
|
||||
|
||||
use crate::common::{RustFSTestEnvironment, init_logging};
|
||||
use aws_sdk_s3::config::{Credentials, Region};
|
||||
use crate::common::{AdminTransport, RustFSTestEnvironment, admin_create_user_via, init_logging};
|
||||
use aws_sdk_s3::Client;
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
use aws_sdk_s3::{Client, Config};
|
||||
use tracing::info;
|
||||
|
||||
/// This suite deliberately drives the admin API through the external `awscurl`
|
||||
/// binary, so user creation pins `AdminTransport::Awscurl`.
|
||||
async fn create_user(
|
||||
env: &RustFSTestEnvironment,
|
||||
username: &str,
|
||||
password: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let create_user_body = serde_json::json!({
|
||||
"secretKey": password,
|
||||
"status": "enabled"
|
||||
})
|
||||
.to_string();
|
||||
|
||||
let create_user_url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username);
|
||||
crate::common::awscurl_put(&create_user_url, &create_user_body, &env.access_key, &env.secret_key).await?;
|
||||
Ok(())
|
||||
admin_create_user_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, username, password).await
|
||||
}
|
||||
|
||||
fn create_user_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
|
||||
let credentials = Credentials::new(access_key, secret_key, None, None, "test-user");
|
||||
let config = Config::builder()
|
||||
.credentials_provider(credentials)
|
||||
.region(Region::new("us-east-1"))
|
||||
.endpoint_url(&env.url)
|
||||
.force_path_style(true)
|
||||
.behavior_version_latest()
|
||||
.build();
|
||||
|
||||
Client::from_conf(config)
|
||||
env.create_s3_client_with_credentials(access_key, secret_key)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
+117
-19
@@ -1744,30 +1744,128 @@ pub(crate) async fn admin_create_user(
|
||||
username: &str,
|
||||
secret_key: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username);
|
||||
let body = serde_json::json!({
|
||||
"secretKey": secret_key,
|
||||
"status": "enabled"
|
||||
});
|
||||
let response = signed_request(
|
||||
http::Method::PUT,
|
||||
&url,
|
||||
&env.access_key,
|
||||
&env.secret_key,
|
||||
Some(body.to_string().into_bytes()),
|
||||
Some("application/json"),
|
||||
)
|
||||
.await?;
|
||||
admin_create_user_via(AdminTransport::Signed, &env.url, &env.access_key, &env.secret_key, username, secret_key).await
|
||||
}
|
||||
|
||||
if response.status() != reqwest::StatusCode::OK {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
return Err(format!("create user failed: {status} {body}").into());
|
||||
/// Transport used by the shared admin-API helpers: in-process SigV4 signing
|
||||
/// via [`signed_request`], or the external `awscurl` binary (an independent
|
||||
/// SigV4 implementation exercised by the awscurl-gated suites).
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum AdminTransport {
|
||||
Signed,
|
||||
Awscurl,
|
||||
}
|
||||
|
||||
/// Execute an admin-API request against `base_url` with admin credentials over
|
||||
/// the chosen transport, failing on any non-success response.
|
||||
pub(crate) async fn admin_execute_at(
|
||||
transport: AdminTransport,
|
||||
method: http::Method,
|
||||
base_url: &str,
|
||||
admin_access_key: &str,
|
||||
admin_secret_key: &str,
|
||||
path_and_query: &str,
|
||||
body: Option<&str>,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let url = format!("{base_url}{path_and_query}");
|
||||
match transport {
|
||||
AdminTransport::Signed => {
|
||||
let content_type = match body {
|
||||
Some(body) if !body.is_empty() => Some("application/json"),
|
||||
_ => None,
|
||||
};
|
||||
let response = signed_request(
|
||||
method.clone(),
|
||||
&url,
|
||||
admin_access_key,
|
||||
admin_secret_key,
|
||||
body.map(|body| body.as_bytes().to_vec()),
|
||||
content_type,
|
||||
)
|
||||
.await?;
|
||||
if !response.status().is_success() {
|
||||
let status = response.status();
|
||||
let text = response.text().await.unwrap_or_default();
|
||||
return Err(format!("{method} {path_and_query} failed: {status} {text}").into());
|
||||
}
|
||||
}
|
||||
AdminTransport::Awscurl => {
|
||||
execute_awscurl(&url, method.as_str(), body, admin_access_key, admin_secret_key).await?;
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Create a new IAM user via the admin API over the chosen transport.
|
||||
pub(crate) async fn admin_create_user_via(
|
||||
transport: AdminTransport,
|
||||
base_url: &str,
|
||||
admin_access_key: &str,
|
||||
admin_secret_key: &str,
|
||||
username: &str,
|
||||
secret_key: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let path = format!("/rustfs/admin/v3/add-user?accessKey={username}");
|
||||
let body = serde_json::json!({"secretKey": secret_key, "status": "enabled"}).to_string();
|
||||
admin_execute_at(
|
||||
transport,
|
||||
http::Method::PUT,
|
||||
base_url,
|
||||
admin_access_key,
|
||||
admin_secret_key,
|
||||
&path,
|
||||
Some(&body),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Install a canned policy via the admin API over the chosen transport.
|
||||
pub(crate) async fn admin_add_canned_policy_via(
|
||||
transport: AdminTransport,
|
||||
base_url: &str,
|
||||
admin_access_key: &str,
|
||||
admin_secret_key: &str,
|
||||
policy_name: &str,
|
||||
policy_json: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let path = format!("/rustfs/admin/v3/add-canned-policy?name={policy_name}");
|
||||
admin_execute_at(
|
||||
transport,
|
||||
http::Method::PUT,
|
||||
base_url,
|
||||
admin_access_key,
|
||||
admin_secret_key,
|
||||
&path,
|
||||
Some(policy_json),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Attach a canned policy to a user via the admin API over the chosen transport.
|
||||
pub(crate) async fn admin_attach_user_policy_via(
|
||||
transport: AdminTransport,
|
||||
base_url: &str,
|
||||
admin_access_key: &str,
|
||||
admin_secret_key: &str,
|
||||
policy_name: &str,
|
||||
username: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let path = format!("/rustfs/admin/v3/set-user-or-group-policy?policyName={policy_name}&userOrGroup={username}&isGroup=false");
|
||||
// `Some("")` preserves the historical wire shape on both transports: awscurl
|
||||
// keeps sending `-d ''` and the signed path attaches an empty body with no
|
||||
// content type.
|
||||
admin_execute_at(
|
||||
transport,
|
||||
http::Method::PUT,
|
||||
base_url,
|
||||
admin_access_key,
|
||||
admin_secret_key,
|
||||
&path,
|
||||
Some(""),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
@@ -16,37 +16,29 @@
|
||||
//! session policy** (`Policy` parameter) via `awscurl --service sts` with explicit
|
||||
//! `Content-Type: application/x-www-form-urlencoded` on `POST /`.
|
||||
|
||||
use crate::common::{RustFSTestEnvironment, awscurl_delete, awscurl_post_sts_form_urlencoded, awscurl_put, init_logging};
|
||||
use aws_sdk_s3::config::{Credentials, Region};
|
||||
use crate::common::{
|
||||
AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via,
|
||||
awscurl_delete, awscurl_post_sts_form_urlencoded, build_test_s3_config, init_logging,
|
||||
};
|
||||
use aws_sdk_s3::Client;
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use aws_sdk_s3::types::{Delete, ObjectIdentifier, Tag, Tagging};
|
||||
use aws_sdk_s3::{Client, Config};
|
||||
use tracing::info;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn user_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
|
||||
let credentials = Credentials::new(access_key, secret_key, None, None, "e2e-existing-tag");
|
||||
let config = Config::builder()
|
||||
.credentials_provider(credentials)
|
||||
.region(Region::new("us-east-1"))
|
||||
.endpoint_url(&env.url)
|
||||
.force_path_style(true)
|
||||
.behavior_version_latest()
|
||||
.build();
|
||||
Client::from_conf(config)
|
||||
env.create_s3_client_with_credentials(access_key, secret_key)
|
||||
}
|
||||
|
||||
fn sts_session_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str, session_token: &str) -> Client {
|
||||
let credentials = Credentials::new(access_key, secret_key, Some(session_token.into()), None, "e2e-sts-session");
|
||||
let config = Config::builder()
|
||||
.credentials_provider(credentials)
|
||||
.region(Region::new("us-east-1"))
|
||||
.endpoint_url(&env.url)
|
||||
.force_path_style(true)
|
||||
.behavior_version_latest()
|
||||
.build();
|
||||
Client::from_conf(config)
|
||||
Client::from_conf(build_test_s3_config(
|
||||
&env.url,
|
||||
access_key,
|
||||
secret_key,
|
||||
Some(session_token),
|
||||
"e2e-sts-session",
|
||||
))
|
||||
}
|
||||
|
||||
fn extract_xml_tag(xml: &str, tag: &str) -> Option<String> {
|
||||
@@ -77,15 +69,16 @@ async fn assume_role_with_session_policy(
|
||||
parse_assume_role_credentials(&xml)
|
||||
}
|
||||
|
||||
// This suite deliberately drives the admin API through the external `awscurl`
|
||||
// binary (an independent SigV4 implementation), so the wrappers below pin
|
||||
// `AdminTransport::Awscurl`.
|
||||
|
||||
async fn admin_create_user(
|
||||
env: &RustFSTestEnvironment,
|
||||
username: &str,
|
||||
password: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let body = serde_json::json!({ "secretKey": password, "status": "enabled" }).to_string();
|
||||
let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username);
|
||||
awscurl_put(&url, &body, &env.access_key, &env.secret_key).await?;
|
||||
Ok(())
|
||||
admin_create_user_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, username, password).await
|
||||
}
|
||||
|
||||
async fn admin_add_canned_policy(
|
||||
@@ -93,9 +86,15 @@ async fn admin_add_canned_policy(
|
||||
policy_name: &str,
|
||||
policy_json: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", env.url, policy_name);
|
||||
awscurl_put(&url, policy_json, &env.access_key, &env.secret_key).await?;
|
||||
Ok(())
|
||||
admin_add_canned_policy_via(
|
||||
AdminTransport::Awscurl,
|
||||
&env.url,
|
||||
&env.access_key,
|
||||
&env.secret_key,
|
||||
policy_name,
|
||||
policy_json,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn admin_attach_policy_to_user(
|
||||
@@ -103,12 +102,7 @@ async fn admin_attach_policy_to_user(
|
||||
policy_name: &str,
|
||||
username: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let url = format!(
|
||||
"{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=false",
|
||||
env.url, policy_name, username
|
||||
);
|
||||
awscurl_put(&url, "", &env.access_key, &env.secret_key).await?;
|
||||
Ok(())
|
||||
admin_attach_user_policy_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, policy_name, username).await
|
||||
}
|
||||
|
||||
async fn admin_remove_user(env: &RustFSTestEnvironment, username: &str) {
|
||||
|
||||
@@ -15,20 +15,11 @@
|
||||
//! E2E tests for group management (fixes #2028).
|
||||
|
||||
use crate::common::{RustFSTestEnvironment, admin_ok, admin_request, init_logging};
|
||||
use aws_sdk_s3::config::{Credentials, Region};
|
||||
use aws_sdk_s3::{Client, Config};
|
||||
use aws_sdk_s3::Client;
|
||||
use tracing::info;
|
||||
|
||||
fn create_user_s3_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
|
||||
let credentials = Credentials::new(access_key, secret_key, None, None, "e2e-group-test");
|
||||
let config = Config::builder()
|
||||
.credentials_provider(credentials)
|
||||
.region(Region::new("us-east-1"))
|
||||
.endpoint_url(&env.url)
|
||||
.force_path_style(true)
|
||||
.behavior_version_latest()
|
||||
.build();
|
||||
Client::from_conf(config)
|
||||
env.create_s3_client_with_credentials(access_key, secret_key)
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
|
||||
@@ -15,13 +15,13 @@
|
||||
//! Four-node EC regression gate for inline storage and the inline GET reader.
|
||||
//!
|
||||
//! The storage decision is based on shard bytes (256 KiB / 32 KiB objects for
|
||||
//! the default EC 2+2 geometry), while the GET fast path has its own object-size
|
||||
//! limits (128 KiB / 16 KiB). A local OTLP/HTTP collector observes the existing
|
||||
//! reader-path counter without adding a scrape endpoint or production logging.
|
||||
//! the default EC 2+2 geometry), and the GET fast path follows the persisted
|
||||
//! inline marker. A local OTLP/HTTP collector observes the existing reader-path
|
||||
//! counter without adding a scrape endpoint or production logging.
|
||||
//! One S3 GET can select readers on multiple EC nodes, so the counter tracks
|
||||
//! distributed reader selection rather than HTTP request count.
|
||||
|
||||
use crate::common::{RustFSTestClusterEnvironment, RustFSTestEnvironment, init_logging, local_http_client};
|
||||
use crate::common::{RustFSTestClusterEnvironment, RustFSTestEnvironment, init_logging};
|
||||
use aws_sdk_s3::Client;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use aws_sdk_s3::types::{
|
||||
@@ -30,7 +30,7 @@ use aws_sdk_s3::types::{
|
||||
};
|
||||
use bytes::Bytes;
|
||||
use flate2::read::GzDecoder;
|
||||
use http::header::{CONTENT_ENCODING, HOST};
|
||||
use http::header::CONTENT_ENCODING;
|
||||
use http::{Method, Request, Response, StatusCode};
|
||||
use http_body_util::{BodyExt, Full};
|
||||
use hyper::body::Incoming;
|
||||
@@ -42,9 +42,6 @@ use opentelemetry_proto::tonic::metrics::v1::{
|
||||
Metric, NumberDataPoint, ResourceMetrics, ScopeMetrics, Sum, metric, number_data_point,
|
||||
};
|
||||
use prost::Message;
|
||||
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
|
||||
use rustfs_signer::sign_v4;
|
||||
use s3s::Body;
|
||||
use std::collections::BTreeMap;
|
||||
use std::convert::Infallible;
|
||||
use std::error::Error;
|
||||
@@ -92,6 +89,7 @@ const MPU_PART_1_SIZE: usize = 5 * 1024 * 1024;
|
||||
const MPU_PART_2_SIZE: usize = 16 * KIB;
|
||||
const TIER_BUCKET: &str = "inline-fallback-cold-tier";
|
||||
const TIER_PREFIX: &str = "tiered";
|
||||
const ALLOW_LOOPBACK_TIER_ENDPOINT_ENV: &str = "RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT";
|
||||
const MSGPACK_FALLBACK_CONTROL_SERIES: [(&str, &str); 4] = [
|
||||
(FALLBACK_REQUEST_DIRECTION, "ReadMultipleReq"),
|
||||
(FALLBACK_RESPONSE_DIRECTION, "ReadMultipleResp"),
|
||||
@@ -794,12 +792,12 @@ fn metric_attribute(key: &str, value: &str) -> KeyValue {
|
||||
}
|
||||
|
||||
fn boundary_cases(state: VersionState) -> Vec<BoundaryCase> {
|
||||
let (fast_limit, storage_limit) = match state {
|
||||
VersionState::Enabled => (16 * KIB, 32 * KIB),
|
||||
VersionState::Unversioned => (128 * KIB, 256 * KIB),
|
||||
let storage_limit = match state {
|
||||
VersionState::Enabled => 32 * KIB,
|
||||
VersionState::Unversioned => 256 * KIB,
|
||||
// A suspended bucket stores its null version using the unversioned
|
||||
// shard threshold, while ObjectInfo keeps version-aware GET semantics.
|
||||
VersionState::Suspended => (16 * KIB, 256 * KIB),
|
||||
VersionState::Suspended => 256 * KIB,
|
||||
};
|
||||
let mut sizes = vec![0, 16 * KIB - 1, 16 * KIB, 16 * KIB + 1, 32 * KIB - 1, 32 * KIB, 32 * KIB + 1];
|
||||
if !matches!(state, VersionState::Enabled) {
|
||||
@@ -820,7 +818,7 @@ fn boundary_cases(state: VersionState) -> Vec<BoundaryCase> {
|
||||
stored_inline: size <= storage_limit,
|
||||
expected_reader_path: if size == 0 {
|
||||
EMPTY
|
||||
} else if size <= fast_limit {
|
||||
} else if size <= storage_limit {
|
||||
INLINE_DIRECT
|
||||
} else {
|
||||
LEGACY_DUPLEX
|
||||
@@ -1262,6 +1260,8 @@ async fn put_two_part_multipart(client: &Client, bucket: &str, key: &str) -> Tes
|
||||
Ok((body, part2, complete.e_tag().map(str::to_owned)))
|
||||
}
|
||||
|
||||
/// Thin wrapper over [`crate::common::admin_request`], kept local so the call
|
||||
/// sites below keep their `Option<&str>` body shape.
|
||||
async fn signed_admin_request(
|
||||
base_url: &str,
|
||||
method: Method,
|
||||
@@ -1270,30 +1270,7 @@ async fn signed_admin_request(
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
) -> TestResult<(reqwest::StatusCode, String)> {
|
||||
let url = format!("{base_url}{path}");
|
||||
let uri = url.parse::<http::Uri>()?;
|
||||
let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
|
||||
let body_bytes = body.map(|value| value.as_bytes().to_vec()).unwrap_or_default();
|
||||
|
||||
let request = http::Request::builder()
|
||||
.method(method.clone())
|
||||
.uri(uri)
|
||||
.header(HOST, authority)
|
||||
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
|
||||
let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
|
||||
|
||||
let client = local_http_client();
|
||||
let mut request_builder = client.request(method, url.as_str());
|
||||
for (name, value) in signed.headers() {
|
||||
request_builder = request_builder.header(name, value);
|
||||
}
|
||||
if !body_bytes.is_empty() {
|
||||
request_builder = request_builder.body(body_bytes);
|
||||
}
|
||||
let response = request_builder.send().await?;
|
||||
let status = response.status();
|
||||
let text = response.text().await?;
|
||||
Ok((status, text))
|
||||
crate::common::admin_request(base_url, method, path, body.map(str::to_string), access_key, secret_key).await
|
||||
}
|
||||
|
||||
fn unique_tier_name() -> String {
|
||||
@@ -2124,6 +2101,7 @@ async fn four_node_add_tier_converges() -> TestResult {
|
||||
cold.create_s3_client().create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
|
||||
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
|
||||
hot.start().await?;
|
||||
|
||||
let tier_name = unique_tier_name();
|
||||
@@ -2142,6 +2120,7 @@ async fn four_node_add_tier_converges_after_offline_node_restart_without_second_
|
||||
cold.create_s3_client().create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
|
||||
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
|
||||
hot.start().await?;
|
||||
|
||||
let tier_name = unique_tier_name();
|
||||
@@ -2238,6 +2217,7 @@ async fn four_node_manual_transition_distributed_admission_conflict_reports_stat
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
|
||||
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
|
||||
hot.set_env("RUSTFS_SCANNER_ENABLED", "false");
|
||||
hot.set_env("RUSTFS_SCANNER_CYCLE", "3600");
|
||||
hot.set_env("RUSTFS_MAX_TRANSITION_WORKERS", "1");
|
||||
@@ -2380,6 +2360,7 @@ async fn four_node_manual_transition_rollout_non_empty_restart_readback() -> Tes
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
|
||||
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
|
||||
hot.set_env("RUSTFS_SCANNER_ENABLED", "false");
|
||||
hot.set_env("RUSTFS_SCANNER_CYCLE", "3600");
|
||||
hot.set_env("RUSTFS_MAX_TRANSITION_WORKERS", "2");
|
||||
@@ -2484,6 +2465,7 @@ async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls_during_
|
||||
|
||||
let collector = OtlpMetricCollector::start().await?;
|
||||
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
|
||||
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
|
||||
configure_mixed_msgpack_cluster(&mut hot, &collector)?;
|
||||
hot.set_env("RUSTFS_SCANNER_CYCLE", "1");
|
||||
hot.set_env("RUSTFS_ILM_PROCESS_TIME", "1");
|
||||
@@ -2595,6 +2577,7 @@ async fn four_node_transitioned_inline_fallback() -> TestResult {
|
||||
|
||||
let collector = OtlpMetricCollector::start().await?;
|
||||
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
|
||||
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
|
||||
configure_reader_metric_cluster(&mut hot, &collector);
|
||||
hot.set_env("RUSTFS_SCANNER_CYCLE", "1");
|
||||
hot.set_env("RUSTFS_ILM_PROCESS_TIME", "1");
|
||||
|
||||
@@ -0,0 +1,220 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Anonymous access to SSE-KMS objects under per-key authorization.
|
||||
//!
|
||||
//! Locks both halves of the anonymous contract decided in backlog#2028 (D4):
|
||||
//!
|
||||
//! - **Enforcement on**: anonymous requests hold no `kms` grants, so a public
|
||||
//! bucket policy does not let them read SSE-KMS objects or write through an
|
||||
//! SSE-KMS default-encryption rule. Both fail with `AccessDenied`.
|
||||
//! - **Enforcement off** (the default): bucket policy alone governs anonymous
|
||||
//! access, matching the pre-enforcement behavior — public SSE-KMS objects are
|
||||
//! decrypted and served, and anonymous writes are encrypted under the default
|
||||
//! key.
|
||||
//!
|
||||
//! The denial today is emergent — an empty-account principal falling through to
|
||||
//! the IAM default deny — so without this file a refactor of principal
|
||||
//! construction or policy evaluation could silently flip it. Each test carries a
|
||||
//! plaintext-object positive control: a denial proves nothing while the bucket
|
||||
//! policy has not propagated.
|
||||
|
||||
use super::common::{LocalKMSTestEnvironment, create_key_with_specific_id};
|
||||
use crate::common::{init_logging, local_http_client};
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use aws_sdk_s3::types::{
|
||||
ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
|
||||
};
|
||||
use std::time::Duration;
|
||||
|
||||
type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
|
||||
|
||||
const DEFAULT_KEY: &str = "kms-anon-default-key";
|
||||
const BUCKET: &str = "kms-anon-enforcement";
|
||||
const PLAIN_OBJECT: &str = "plain.txt";
|
||||
const ENCRYPTED_OBJECT: &str = "encrypted.txt";
|
||||
const PAYLOAD: &[u8] = b"kms anonymous enforcement payload";
|
||||
|
||||
/// How long a bucket policy change may take to reach the request path.
|
||||
const POLICY_PROPAGATION: Duration = Duration::from_secs(20);
|
||||
|
||||
/// Start a local-KMS server and build the public-bucket fixture.
|
||||
///
|
||||
/// The bucket holds a plaintext object (the positive control), an SSE-KMS
|
||||
/// object, an SSE-KMS default-encryption rule, and a bucket policy opening
|
||||
/// `GetObject`/`PutObject` to everyone. The enforcement switch defaults to off,
|
||||
/// so the enforcing case has to set it explicitly.
|
||||
async fn start_public_sse_kms_bucket(env: &mut LocalKMSTestEnvironment, enforce: bool) -> TestResult {
|
||||
create_key_with_specific_id(&env.kms_keys_dir, DEFAULT_KEY).await?;
|
||||
|
||||
let key_dir = env.kms_keys_dir.clone();
|
||||
let args = vec![
|
||||
"--kms-enable",
|
||||
"--kms-backend",
|
||||
"local",
|
||||
"--kms-key-dir",
|
||||
key_dir.as_str(),
|
||||
"--kms-default-key-id",
|
||||
DEFAULT_KEY,
|
||||
];
|
||||
let mut envs = vec![("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")];
|
||||
if enforce {
|
||||
envs.push(("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "true"));
|
||||
}
|
||||
env.base_env.start_rustfs_server_with_env(args, &envs).await?;
|
||||
env.base_env.create_test_bucket(BUCKET).await?;
|
||||
|
||||
let owner = env.base_env.create_s3_client();
|
||||
|
||||
owner
|
||||
.put_object()
|
||||
.bucket(BUCKET)
|
||||
.key(PLAIN_OBJECT)
|
||||
.body(ByteStream::from_static(PAYLOAD))
|
||||
.send()
|
||||
.await?;
|
||||
owner
|
||||
.put_object()
|
||||
.bucket(BUCKET)
|
||||
.key(ENCRYPTED_OBJECT)
|
||||
.body(ByteStream::from_static(PAYLOAD))
|
||||
.server_side_encryption(ServerSideEncryption::AwsKms)
|
||||
.ssekms_key_id(DEFAULT_KEY)
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
let encryption_config = ServerSideEncryptionConfiguration::builder()
|
||||
.rules(
|
||||
ServerSideEncryptionRule::builder()
|
||||
.apply_server_side_encryption_by_default(
|
||||
ServerSideEncryptionByDefault::builder()
|
||||
.sse_algorithm(ServerSideEncryption::AwsKms)
|
||||
.kms_master_key_id(DEFAULT_KEY)
|
||||
.build()?,
|
||||
)
|
||||
.build(),
|
||||
)
|
||||
.build()?;
|
||||
owner
|
||||
.put_bucket_encryption()
|
||||
.bucket(BUCKET)
|
||||
.server_side_encryption_configuration(encryption_config)
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
let policy = serde_json::json!({
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [{
|
||||
"Sid": "PublicReadWrite",
|
||||
"Effect": "Allow",
|
||||
"Principal": "*",
|
||||
"Action": ["s3:GetObject", "s3:PutObject"],
|
||||
"Resource": [format!("arn:aws:s3:::{BUCKET}/*")]
|
||||
}]
|
||||
})
|
||||
.to_string();
|
||||
owner.put_bucket_policy().bucket(BUCKET).policy(&policy).send().await?;
|
||||
let _ = owner.delete_public_access_block().bucket(BUCKET).send().await;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn object_url(env: &LocalKMSTestEnvironment, key: &str) -> String {
|
||||
format!("{}/{BUCKET}/{key}", env.base_env.url)
|
||||
}
|
||||
|
||||
async fn anonymous_get(env: &LocalKMSTestEnvironment, key: &str) -> Result<reqwest::Response, reqwest::Error> {
|
||||
local_http_client().get(object_url(env, key)).send().await
|
||||
}
|
||||
|
||||
async fn anonymous_put(env: &LocalKMSTestEnvironment, key: &str) -> Result<reqwest::Response, reqwest::Error> {
|
||||
local_http_client().put(object_url(env, key)).body(PAYLOAD).send().await
|
||||
}
|
||||
|
||||
/// Retry the plaintext read until the public bucket policy is live.
|
||||
async fn wait_for_public_read(env: &LocalKMSTestEnvironment) -> TestResult {
|
||||
let deadline = tokio::time::Instant::now() + POLICY_PROPAGATION;
|
||||
loop {
|
||||
let status = anonymous_get(env, PLAIN_OBJECT).await?.status();
|
||||
if status.as_u16() == 200 {
|
||||
return Ok(());
|
||||
}
|
||||
if tokio::time::Instant::now() >= deadline {
|
||||
return Err(format!("positive control never became readable: anonymous GET {PLAIN_OBJECT} -> {status}").into());
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||
}
|
||||
}
|
||||
|
||||
async fn assert_anonymous_denied(response: reqwest::Response, what: &str) -> TestResult {
|
||||
let status = response.status().as_u16();
|
||||
let body = response.text().await?;
|
||||
assert_eq!(status, 403, "{what} must be denied, got {status}: {body}");
|
||||
assert!(body.contains("AccessDenied"), "{what} must carry AccessDenied: {body}");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Enforcement on: a public bucket policy does not exempt anonymous requests
|
||||
/// from per-key authorization, on either the read or the default-encryption
|
||||
/// write path.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn anonymous_sse_kms_denied_under_enforcement() -> TestResult {
|
||||
init_logging();
|
||||
|
||||
let mut env = LocalKMSTestEnvironment::new().await?;
|
||||
start_public_sse_kms_bucket(&mut env, true).await?;
|
||||
wait_for_public_read(&env).await?;
|
||||
|
||||
let read = anonymous_get(&env, ENCRYPTED_OBJECT).await?;
|
||||
assert_anonymous_denied(read, "anonymous GET of an SSE-KMS object").await?;
|
||||
|
||||
let write = anonymous_put(&env, "anon-write.txt").await?;
|
||||
assert_anonymous_denied(write, "anonymous PUT through an SSE-KMS default-encryption rule").await?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Enforcement off (the default): bucket policy alone governs anonymous access,
|
||||
/// and the default-encryption rule still encrypts anonymous writes.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn anonymous_sse_kms_governed_by_bucket_policy_without_enforcement() -> TestResult {
|
||||
init_logging();
|
||||
|
||||
let mut env = LocalKMSTestEnvironment::new().await?;
|
||||
start_public_sse_kms_bucket(&mut env, false).await?;
|
||||
wait_for_public_read(&env).await?;
|
||||
|
||||
let read = anonymous_get(&env, ENCRYPTED_OBJECT).await?;
|
||||
assert_eq!(read.status().as_u16(), 200, "anonymous GET of a public SSE-KMS object must succeed");
|
||||
assert_eq!(read.bytes().await?.as_ref(), PAYLOAD, "the object must be served decrypted");
|
||||
|
||||
let write = anonymous_put(&env, "anon-write.txt").await?;
|
||||
assert_eq!(write.status().as_u16(), 200, "anonymous PUT to a public bucket must succeed");
|
||||
|
||||
let stored = env
|
||||
.base_env
|
||||
.create_s3_client()
|
||||
.head_object()
|
||||
.bucket(BUCKET)
|
||||
.key("anon-write.txt")
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(
|
||||
stored.server_side_encryption(),
|
||||
Some(&ServerSideEncryption::AwsKms),
|
||||
"the anonymous write must be encrypted by the bucket default rule"
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -66,6 +66,7 @@ const SURVIVOR_KEY: &str = "keep/object.bin";
|
||||
const TIER_NAME: &str = "KMSCOLD";
|
||||
const TIER_BUCKET: &str = "kms-ilm-cold-tier";
|
||||
const TIER_PREFIX: &str = "tiered";
|
||||
const ALLOW_LOOPBACK_TIER_ENDPOINT_ENV: (&str, &str) = ("RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT", "true");
|
||||
const TRANSITION_BUCKET: &str = "kms-ilm-transition";
|
||||
const TRANSITION_KEY: &str = "tier/object.bin";
|
||||
|
||||
@@ -80,7 +81,7 @@ const ILM_DEADLINE: StdDuration = StdDuration::from_secs(90);
|
||||
/// `--kms-default-key-id`, insecure dev defaults). The lifecycle env matches
|
||||
/// `reliant/lifecycle.rs::fast_lifecycle_env` plus `RUSTFS_ILM_DEBUG_DAY_SECS=2`,
|
||||
/// so a `Days=1` rule is due about two seconds after the write.
|
||||
async fn start_enforcing_ilm_server(env: &mut LocalKMSTestEnvironment) -> TestResult {
|
||||
async fn start_enforcing_ilm_server(env: &mut LocalKMSTestEnvironment, extra_env: &[(&str, &str)]) -> TestResult {
|
||||
create_key_with_specific_id(&env.kms_keys_dir, SSE_KEY).await?;
|
||||
|
||||
let key_dir = env.kms_keys_dir.clone();
|
||||
@@ -94,13 +95,14 @@ async fn start_enforcing_ilm_server(env: &mut LocalKMSTestEnvironment) -> TestRe
|
||||
SSE_KEY,
|
||||
];
|
||||
|
||||
let envs = [
|
||||
let mut envs = vec![
|
||||
("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true"),
|
||||
("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "true"),
|
||||
("RUSTFS_SCANNER_CYCLE", "1"),
|
||||
("RUSTFS_ILM_PROCESS_TIME", "1"),
|
||||
("RUSTFS_ILM_DEBUG_DAY_SECS", "2"),
|
||||
];
|
||||
envs.extend_from_slice(extra_env);
|
||||
|
||||
env.base_env.start_rustfs_server_with_env(args, &envs).await?;
|
||||
Ok(())
|
||||
@@ -427,7 +429,7 @@ async fn ilm_expiration_on_sse_kms_bucket_under_enforcement() -> TestResult {
|
||||
init_logging();
|
||||
|
||||
let mut env = LocalKMSTestEnvironment::new().await?;
|
||||
start_enforcing_ilm_server(&mut env).await?;
|
||||
start_enforcing_ilm_server(&mut env, &[]).await?;
|
||||
env.base_env.create_test_bucket(EXPIRY_BUCKET).await?;
|
||||
|
||||
let client = env.base_env.create_s3_client();
|
||||
@@ -499,7 +501,7 @@ async fn ilm_transition_on_sse_kms_bucket_under_enforcement_reads_back() -> Test
|
||||
|
||||
// Hot server: Local KMS + enforcement + accelerated lifecycle clock.
|
||||
let mut env = LocalKMSTestEnvironment::new().await?;
|
||||
start_enforcing_ilm_server(&mut env).await?;
|
||||
start_enforcing_ilm_server(&mut env, &[ALLOW_LOOPBACK_TIER_ENDPOINT_ENV]).await?;
|
||||
let hot_client = env.base_env.create_s3_client();
|
||||
|
||||
add_rustfs_tier(&env.base_env, &cold.base_env).await?;
|
||||
|
||||
@@ -57,6 +57,9 @@ mod copy_object_version_restore_sse_test;
|
||||
#[cfg(test)]
|
||||
mod configured_roundtrip_test;
|
||||
|
||||
#[cfg(test)]
|
||||
mod kms_anonymous_enforcement_test;
|
||||
|
||||
#[cfg(test)]
|
||||
mod kms_authorization_negative_matrix_test;
|
||||
|
||||
|
||||
@@ -38,14 +38,10 @@ use aws_sdk_s3::types::{
|
||||
NotificationConfiguration, NotificationConfigurationFilter, ObjectIdentifier, QueueConfiguration, S3KeyFilter,
|
||||
VersioningConfiguration,
|
||||
};
|
||||
use http::header::{CONTENT_TYPE, HOST};
|
||||
use local_ip_address::local_ip;
|
||||
use reqwest::StatusCode;
|
||||
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
|
||||
use rustfs_signer::sign_v4;
|
||||
use rustfs_utils::egress::ENV_OUTBOUND_ALLOW_ORIGINS;
|
||||
use rustfs_utils::http::headers::{AMZ_REQUEST_ID, REQUEST_ID_HEADER};
|
||||
use s3s::Body;
|
||||
use serde_json::Value;
|
||||
use std::error::Error;
|
||||
use std::io::Cursor;
|
||||
@@ -415,42 +411,16 @@ async fn collect_until(
|
||||
// Admin target configuration (signed admin HTTP)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Thin wrapper over [`crate::common::signed_request`] with this suite's
|
||||
/// root credentials; a `Some` body is always JSON here.
|
||||
async fn signed_admin_request(
|
||||
env: &RustFSTestEnvironment,
|
||||
method: http::Method,
|
||||
url: &str,
|
||||
body: Option<Vec<u8>>,
|
||||
) -> Result<reqwest::Response, BoxError> {
|
||||
let uri = url.parse::<http::Uri>()?;
|
||||
let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
|
||||
let mut builder = http::Request::builder()
|
||||
.method(method.clone())
|
||||
.uri(uri)
|
||||
.header(HOST, authority)
|
||||
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
|
||||
if body.is_some() {
|
||||
builder = builder.header(CONTENT_TYPE, "application/json");
|
||||
}
|
||||
|
||||
let content_len = body.as_ref().map(|b| b.len() as i64).unwrap_or_default();
|
||||
let signed = sign_v4(
|
||||
builder.body(Body::empty())?,
|
||||
content_len,
|
||||
&env.access_key,
|
||||
&env.secret_key,
|
||||
"",
|
||||
"us-east-1",
|
||||
);
|
||||
|
||||
let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
|
||||
let mut request = crate::common::local_http_client().request(reqwest_method, url);
|
||||
for (name, value) in signed.headers() {
|
||||
request = request.header(name, value);
|
||||
}
|
||||
if let Some(body) = body {
|
||||
request = request.body(body);
|
||||
}
|
||||
Ok(request.send().await?)
|
||||
let content_type = body.is_some().then_some("application/json");
|
||||
crate::common::signed_request(method, url, &env.access_key, &env.secret_key, body, content_type).await
|
||||
}
|
||||
|
||||
async fn enable_notify_module(env: &RustFSTestEnvironment) -> TestResult {
|
||||
|
||||
@@ -15,28 +15,24 @@
|
||||
//! Tests for AWS IAM policy variables with single-value, multi-value, and nested scenarios
|
||||
|
||||
use crate::common::{
|
||||
RustFSTestEnvironment, awscurl_delete, awscurl_put, build_test_s3_config, build_test_sts_client, init_logging,
|
||||
AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via,
|
||||
awscurl_delete, awscurl_put, build_test_s3_config, build_test_sts_client, init_logging,
|
||||
};
|
||||
use aws_sdk_s3::Client;
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use tracing::info;
|
||||
|
||||
/// Helper function to create a regular user with given credentials
|
||||
/// Helper function to create a regular user with given credentials.
|
||||
///
|
||||
/// This suite deliberately drives the admin API through the external `awscurl`
|
||||
/// binary, so the shared helpers are pinned to `AdminTransport::Awscurl`.
|
||||
async fn create_user(
|
||||
env: &RustFSTestEnvironment,
|
||||
username: &str,
|
||||
password: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let create_user_body = serde_json::json!({
|
||||
"secretKey": password,
|
||||
"status": "enabled"
|
||||
})
|
||||
.to_string();
|
||||
|
||||
let create_user_url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username);
|
||||
awscurl_put(&create_user_url, &create_user_body, &env.access_key, &env.secret_key).await?;
|
||||
Ok(())
|
||||
admin_create_user_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, username, password).await
|
||||
}
|
||||
|
||||
/// Helper function to create and attach a policy
|
||||
@@ -46,18 +42,17 @@ async fn create_and_attach_policy(
|
||||
username: &str,
|
||||
policy_document: serde_json::Value,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let policy_string = policy_document.to_string();
|
||||
|
||||
// Create policy
|
||||
let add_policy_url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", env.url, policy_name);
|
||||
awscurl_put(&add_policy_url, &policy_string, &env.access_key, &env.secret_key).await?;
|
||||
|
||||
// Attach policy to user
|
||||
let attach_policy_url = format!(
|
||||
"{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=false",
|
||||
env.url, policy_name, username
|
||||
);
|
||||
awscurl_put(&attach_policy_url, "", &env.access_key, &env.secret_key).await?;
|
||||
admin_add_canned_policy_via(
|
||||
AdminTransport::Awscurl,
|
||||
&env.url,
|
||||
&env.access_key,
|
||||
&env.secret_key,
|
||||
policy_name,
|
||||
&policy_document.to_string(),
|
||||
)
|
||||
.await?;
|
||||
admin_attach_user_policy_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, policy_name, username)
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
@@ -31,15 +31,11 @@
|
||||
//!
|
||||
//! Advisory: <https://github.com/rustfs/rustfs/security/advisories/GHSA-3p3x-734c-h5vx>
|
||||
|
||||
use crate::common::local_http_client;
|
||||
use crate::common::rustfs_binary_path_with_features;
|
||||
use crate::common::{AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via};
|
||||
use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ProtocolTestEnvironment};
|
||||
use anyhow::Result;
|
||||
use http::header::{CONTENT_TYPE, HOST};
|
||||
use reqwest::Client;
|
||||
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
|
||||
use rustfs_signer::sign_v4;
|
||||
use s3s::Body;
|
||||
use tokio::process::Command;
|
||||
use tracing::info;
|
||||
|
||||
@@ -67,92 +63,43 @@ fn basic_auth_header_for(access_key: &str, secret_key: &str) -> String {
|
||||
format!("Basic {}", encoded)
|
||||
}
|
||||
|
||||
async fn signed_admin_request(
|
||||
method: http::Method,
|
||||
url: &str,
|
||||
body: Option<Vec<u8>>,
|
||||
content_type: Option<&str>,
|
||||
) -> Result<reqwest::Response> {
|
||||
let uri = url.parse::<http::Uri>()?;
|
||||
let authority = uri
|
||||
.authority()
|
||||
.ok_or_else(|| anyhow::anyhow!("request URL missing authority"))?
|
||||
.to_string();
|
||||
let mut request = http::Request::builder().method(method.clone()).uri(uri);
|
||||
request = request.header(HOST, authority);
|
||||
request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
|
||||
if let Some(content_type) = content_type {
|
||||
request = request.header(CONTENT_TYPE, content_type);
|
||||
}
|
||||
|
||||
let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default();
|
||||
let signed = sign_v4(
|
||||
request.body(Body::empty())?,
|
||||
content_len,
|
||||
async fn admin_create_user(base_url: &str, username: &str, secret_key: &str) -> Result<()> {
|
||||
admin_create_user_via(
|
||||
AdminTransport::Signed,
|
||||
base_url,
|
||||
DEFAULT_ACCESS_KEY,
|
||||
DEFAULT_SECRET_KEY,
|
||||
"",
|
||||
"us-east-1",
|
||||
);
|
||||
|
||||
let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
|
||||
let mut request_builder = local_http_client().request(reqwest_method, url);
|
||||
for (name, value) in signed.headers() {
|
||||
request_builder = request_builder.header(name, value);
|
||||
}
|
||||
if let Some(body) = body {
|
||||
request_builder = request_builder.body(body);
|
||||
}
|
||||
|
||||
Ok(request_builder.send().await?)
|
||||
}
|
||||
|
||||
async fn admin_create_user(base_url: &str, username: &str, secret_key: &str) -> Result<()> {
|
||||
let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", base_url, username);
|
||||
let body = serde_json::json!({
|
||||
"secretKey": secret_key,
|
||||
"status": "enabled"
|
||||
});
|
||||
let response =
|
||||
signed_admin_request(http::Method::PUT, &url, Some(body.to_string().into_bytes()), Some("application/json")).await?;
|
||||
|
||||
if response.status() != reqwest::StatusCode::OK {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
anyhow::bail!("create user failed: {status} {body}");
|
||||
}
|
||||
|
||||
Ok(())
|
||||
username,
|
||||
secret_key,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!(e))
|
||||
}
|
||||
|
||||
async fn admin_add_canned_policy(base_url: &str, policy_name: &str, policy: &serde_json::Value) -> Result<()> {
|
||||
let url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", base_url, policy_name);
|
||||
let response =
|
||||
signed_admin_request(http::Method::PUT, &url, Some(policy.to_string().into_bytes()), Some("application/json")).await?;
|
||||
|
||||
if response.status() != reqwest::StatusCode::OK {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
anyhow::bail!("add canned policy failed: {status} {body}");
|
||||
}
|
||||
|
||||
Ok(())
|
||||
admin_add_canned_policy_via(
|
||||
AdminTransport::Signed,
|
||||
base_url,
|
||||
DEFAULT_ACCESS_KEY,
|
||||
DEFAULT_SECRET_KEY,
|
||||
policy_name,
|
||||
&policy.to_string(),
|
||||
)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!(e))
|
||||
}
|
||||
|
||||
async fn admin_attach_policy_to_user(base_url: &str, policy_name: &str, username: &str) -> Result<()> {
|
||||
let url = format!(
|
||||
"{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=false",
|
||||
base_url, policy_name, username
|
||||
);
|
||||
let response = signed_admin_request(http::Method::PUT, &url, Some(Vec::new()), None).await?;
|
||||
|
||||
if response.status() != reqwest::StatusCode::OK {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
anyhow::bail!("attach policy failed: {status} {body}");
|
||||
}
|
||||
|
||||
Ok(())
|
||||
admin_attach_user_policy_via(
|
||||
AdminTransport::Signed,
|
||||
base_url,
|
||||
DEFAULT_ACCESS_KEY,
|
||||
DEFAULT_SECRET_KEY,
|
||||
policy_name,
|
||||
username,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!(e))
|
||||
}
|
||||
|
||||
/// Test WebDAV: MKCOL (create bucket), PUT, GET, DELETE, PROPFIND operations
|
||||
|
||||
@@ -23,9 +23,9 @@
|
||||
//!
|
||||
//! There are no containers, no external S3 backend and no `awscurl`: the
|
||||
//! `AddTier` admin call is signed in-process with `rustfs_signer`, exactly like
|
||||
//! the other admin-API e2e suites in this crate. The RustFS warm backend has no
|
||||
//! loopback/SSRF restriction (that guard is replication-only), so `hot` can tier
|
||||
//! to `cold` over `http://127.0.0.1:<port>`.
|
||||
//! the other admin-API e2e suites in this crate. The source server uses the
|
||||
//! explicit test-only loopback opt-in to tier to `cold` over
|
||||
//! `http://127.0.0.1:<port>` while production keeps the SSRF guard enabled.
|
||||
//!
|
||||
//! The hermetic tests drive the transition and restore paths and pin the
|
||||
//! chains required by ilm-7 and the restore follow-up:
|
||||
@@ -46,7 +46,7 @@
|
||||
//! retry serves the object locally until expiry, and expiry leaves the
|
||||
//! remote object available for a second restore.
|
||||
|
||||
use crate::common::{RustFSTestEnvironment, local_http_client};
|
||||
use crate::common::RustFSTestEnvironment;
|
||||
use aws_sdk_s3::Client;
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
@@ -56,10 +56,6 @@ use aws_sdk_s3::types::{
|
||||
VersioningConfiguration,
|
||||
};
|
||||
use http::Method;
|
||||
use http::header::HOST;
|
||||
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
|
||||
use rustfs_signer::sign_v4;
|
||||
use s3s::Body;
|
||||
use serde::Deserialize;
|
||||
use std::time::{Duration as StdDuration, Instant};
|
||||
use time::{OffsetDateTime, format_description::well_known::Rfc3339};
|
||||
@@ -100,6 +96,7 @@ const MANUAL_ACTIVE_CANCEL_OBJECTS: usize = 512;
|
||||
const MANUAL_RESTART_CANCEL_OBJECTS: usize = 512;
|
||||
const MANUAL_ACTIVE_CANCEL_RUNNING_TIMEOUT: StdDuration = StdDuration::from_secs(15);
|
||||
const MANUAL_TRANSITION_CANCEL_BARRIER_ENV: &str = "RUSTFS_E2E_MANUAL_TRANSITION_CANCEL_BARRIER";
|
||||
const ALLOW_LOOPBACK_TIER_ENDPOINT_ENV: (&str, &str) = ("RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT", "true");
|
||||
const MANUAL_ASYNC_CONFLICT_TERMINAL_TIMEOUT: StdDuration = StdDuration::from_secs(90);
|
||||
const MANUAL_RESTART_RECOVERY_TIMEOUT: StdDuration = StdDuration::from_secs(80);
|
||||
const OBJECT_KEY: &str = "tier/鲁A12345/report.bin";
|
||||
@@ -116,6 +113,20 @@ const HDR_SOURCE_REPLICATION_REQUEST: &str = "x-rustfs-source-replication-reques
|
||||
const HDR_SOURCE_MTIME: &str = "x-rustfs-source-mtime";
|
||||
const TIER_MUTATION_RECOVERY_CHANGED: &str = "Remote tier mutation recovery changed before publish";
|
||||
|
||||
async fn start_tier_source(hot: &mut RustFSTestEnvironment, extra_env: &[(&str, &str)]) -> TestResult {
|
||||
let mut env = Vec::with_capacity(extra_env.len() + 1);
|
||||
env.push(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV);
|
||||
env.extend_from_slice(extra_env);
|
||||
hot.start_rustfs_server_with_env(vec![], &env).await
|
||||
}
|
||||
|
||||
async fn restart_tier_source(hot: &mut RustFSTestEnvironment, extra_env: &[(&str, &str)]) -> TestResult {
|
||||
let mut env = Vec::with_capacity(extra_env.len() + 1);
|
||||
env.push(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV);
|
||||
env.extend_from_slice(extra_env);
|
||||
hot.restart_server_preserving_data(vec![], &env).await
|
||||
}
|
||||
|
||||
/// 5 MiB — the S3 minimum size for a non-final multipart part; the object's only
|
||||
/// internal part boundary sits at this offset.
|
||||
const PART0_SIZE: usize = 5 * 1024 * 1024;
|
||||
@@ -131,9 +142,8 @@ fn payload() -> Vec<u8> {
|
||||
|
||||
/// Sign and send an admin request in-process (no `awscurl`).
|
||||
///
|
||||
/// Mirrors the shared admin-API e2e pattern: the SigV4 signature is computed
|
||||
/// over `UNSIGNED_PAYLOAD`, so the JSON body rides on the wire without being
|
||||
/// pre-hashed. Returns the response status and body text.
|
||||
/// Thin wrapper over [`crate::common::admin_request`], kept local so the call
|
||||
/// sites below keep their `Option<&str>` body shape.
|
||||
async fn signed_admin_request(
|
||||
base_url: &str,
|
||||
method: Method,
|
||||
@@ -142,30 +152,7 @@ async fn signed_admin_request(
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
) -> Result<(reqwest::StatusCode, String), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let url = format!("{base_url}{path}");
|
||||
let uri = url.parse::<http::Uri>()?;
|
||||
let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
|
||||
let body_bytes = body.map(|b| b.as_bytes().to_vec()).unwrap_or_default();
|
||||
|
||||
let request = http::Request::builder()
|
||||
.method(method.clone())
|
||||
.uri(uri)
|
||||
.header(HOST, authority)
|
||||
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
|
||||
let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
|
||||
|
||||
let client = local_http_client();
|
||||
let mut request_builder = client.request(method, url.as_str());
|
||||
for (name, value) in signed.headers() {
|
||||
request_builder = request_builder.header(name, value);
|
||||
}
|
||||
if !body_bytes.is_empty() {
|
||||
request_builder = request_builder.body(body_bytes);
|
||||
}
|
||||
let response = request_builder.send().await?;
|
||||
let status = response.status();
|
||||
let text = response.text().await?;
|
||||
Ok((status, text))
|
||||
crate::common::admin_request(base_url, method, path, body.map(str::to_string), access_key, secret_key).await
|
||||
}
|
||||
|
||||
/// Wire `hot` -> `cold` as a `TierType::RustFS` remote tier via `AddTier`.
|
||||
@@ -888,8 +875,7 @@ async fn test_hermetic_transition_main_path() -> TestResult {
|
||||
// Hot/source server. A 1s scanner cycle is a backstop; transition is
|
||||
// primarily driven immediately by the multipart completion path.
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_CYCLE", "1")])
|
||||
.await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_CYCLE", "1")]).await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
|
||||
// Wire the RustFS remote tier (real connectivity probe, no force).
|
||||
@@ -987,8 +973,7 @@ async fn test_hermetic_transition_restore_failure_expiry_and_retry() -> TestResu
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_ILM_DEBUG_DAY_SECS", "5")])
|
||||
.await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_ILM_DEBUG_DAY_SECS", "5")]).await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -1116,8 +1101,7 @@ async fn test_manual_transition_run_black_box_semantics() -> TestResult {
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
let due_mtime = OffsetDateTime::now_utc() - time::Duration::hours(25);
|
||||
@@ -1222,8 +1206,7 @@ async fn test_manual_transition_async_job_status_polling() -> TestResult {
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -1321,8 +1304,7 @@ async fn test_manual_transition_async_limit_reports_terminal_partial() -> TestRe
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -1483,8 +1465,8 @@ async fn test_manual_transition_async_scope_conflicts_report_active_job() -> Tes
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
hot.start_rustfs_server_with_env(
|
||||
vec![],
|
||||
start_tier_source(
|
||||
&mut hot,
|
||||
&[
|
||||
("RUSTFS_SCANNER_ENABLED", "false"),
|
||||
("RUSTFS_SCANNER_CYCLE", "3600"),
|
||||
@@ -1593,8 +1575,7 @@ async fn test_manual_transition_async_different_buckets_admit_concurrently() ->
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -1714,8 +1695,7 @@ async fn test_manual_transition_async_tier_failure_reports_terminal_partial() ->
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -1807,8 +1787,7 @@ async fn test_manual_transition_async_worker_failure_reports_terminal_partial()
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
cold.stop_server();
|
||||
@@ -1900,8 +1879,8 @@ async fn test_manual_transition_async_active_cancel_reports_terminal_cancelled()
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
hot.start_rustfs_server_with_env(
|
||||
vec![],
|
||||
start_tier_source(
|
||||
&mut hot,
|
||||
&[
|
||||
("RUSTFS_SCANNER_ENABLED", "false"),
|
||||
("RUSTFS_SCANNER_CYCLE", "3600"),
|
||||
@@ -2006,7 +1985,7 @@ async fn test_manual_transition_async_cancel_after_process_restart_recovers_term
|
||||
("RUSTFS_TRANSITION_QUEUE_CAPACITY", "512"),
|
||||
];
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &restart_env).await?;
|
||||
start_tier_source(&mut hot, &restart_env).await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -2034,7 +2013,7 @@ async fn test_manual_transition_async_cancel_after_process_restart_recovers_term
|
||||
.ok_or("async response must include status_endpoint")?;
|
||||
assert_eq!(accepted.cancel_endpoint.as_deref(), Some(status_endpoint));
|
||||
|
||||
hot.restart_server_preserving_data(vec![], &restart_env).await?;
|
||||
restart_tier_source(&mut hot, &restart_env).await?;
|
||||
|
||||
let restarted = manual_transition_job_status(&hot, status_endpoint).await?;
|
||||
assert_eq!(restarted.job_id, job_id);
|
||||
@@ -2158,8 +2137,7 @@ async fn test_manual_transition_run_contract_no_status_cancel_fields() -> TestRe
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -2197,8 +2175,7 @@ async fn test_manual_transition_run_continuation_token_resumes_without_raw_marke
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -2238,8 +2215,7 @@ async fn test_manual_transition_run_continuation_token_resumes_without_raw_marke
|
||||
"continuation token must not expose the raw object prefix: {continuation}"
|
||||
);
|
||||
|
||||
hot.restart_server_preserving_data(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
restart_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
|
||||
let second = manual_transition_run_with_max_and_continuation(
|
||||
&hot,
|
||||
@@ -2272,8 +2248,8 @@ async fn test_manual_transition_run_queue_pressure_partial() -> TestResult {
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
hot.start_rustfs_server_with_env(
|
||||
vec![],
|
||||
start_tier_source(
|
||||
&mut hot,
|
||||
&[
|
||||
("RUSTFS_SCANNER_ENABLED", "false"),
|
||||
("RUSTFS_SCANNER_CYCLE", "3600"),
|
||||
|
||||
@@ -13,8 +13,9 @@
|
||||
// limitations under the License.
|
||||
|
||||
use crate::common::{
|
||||
RustFSTestEnvironment, admin_create_user, awscurl_post_sts_form_urlencoded, init_logging, local_http_client,
|
||||
replication_fast_env, rustfs_binary_path, signed_request, signed_request_with_client, signed_request_with_session_token,
|
||||
AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user,
|
||||
awscurl_post_sts_form_urlencoded, init_logging, local_http_client, replication_fast_env, rustfs_binary_path, signed_request,
|
||||
signed_request_with_client, signed_request_with_session_token,
|
||||
};
|
||||
use crate::fake_s3_target::{
|
||||
FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction as FakeTargetFault, Operation as FakeTargetOperation,
|
||||
@@ -25,7 +26,7 @@ use crate::kms::common::{
|
||||
sse_customer_key_md5_base64,
|
||||
};
|
||||
use crate::storage_api::replication_extension::BucketTargetSys;
|
||||
use aws_sdk_s3::config::{Credentials, Region};
|
||||
use aws_sdk_s3::Client;
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
use aws_sdk_s3::operation::list_object_versions::ListObjectVersionsOutput;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
@@ -33,7 +34,6 @@ use aws_sdk_s3::types::{
|
||||
BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, DeleteMarkerEntry, ObjectVersion, ServerSideEncryption,
|
||||
VersioningConfiguration,
|
||||
};
|
||||
use aws_sdk_s3::{Client, Config};
|
||||
use base64_simd::STANDARD as BASE64_STANDARD;
|
||||
use bytes::Bytes;
|
||||
use flate2::read::GzDecoder;
|
||||
@@ -895,15 +895,7 @@ async fn wait_for_replicated_object_over_https(
|
||||
}
|
||||
|
||||
fn create_user_s3_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
|
||||
let credentials = Credentials::new(access_key, secret_key, None, None, "e2e-site-replication");
|
||||
let config = Config::builder()
|
||||
.credentials_provider(credentials)
|
||||
.region(Region::new("us-east-1"))
|
||||
.endpoint_url(&env.url)
|
||||
.force_path_style(true)
|
||||
.behavior_version_latest()
|
||||
.build();
|
||||
Client::from_conf(config)
|
||||
env.create_s3_client_with_credentials(access_key, secret_key)
|
||||
}
|
||||
|
||||
async fn admin_add_canned_policy(
|
||||
@@ -911,24 +903,15 @@ async fn admin_add_canned_policy(
|
||||
policy_name: &str,
|
||||
policy: &serde_json::Value,
|
||||
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
let url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", env.url, policy_name);
|
||||
let response = signed_request(
|
||||
http::Method::PUT,
|
||||
&url,
|
||||
admin_add_canned_policy_via(
|
||||
AdminTransport::Signed,
|
||||
&env.url,
|
||||
&env.access_key,
|
||||
&env.secret_key,
|
||||
Some(policy.to_string().into_bytes()),
|
||||
Some("application/json"),
|
||||
policy_name,
|
||||
&policy.to_string(),
|
||||
)
|
||||
.await?;
|
||||
|
||||
if response.status() != StatusCode::OK {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
return Err(format!("add canned policy failed: {status} {body}").into());
|
||||
}
|
||||
|
||||
Ok(())
|
||||
.await
|
||||
}
|
||||
|
||||
async fn admin_attach_policy_to_user(
|
||||
@@ -936,19 +919,7 @@ async fn admin_attach_policy_to_user(
|
||||
policy_name: &str,
|
||||
username: &str,
|
||||
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
let url = format!(
|
||||
"{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=false",
|
||||
env.url, policy_name, username
|
||||
);
|
||||
let response = signed_request(http::Method::PUT, &url, &env.access_key, &env.secret_key, Some(Vec::new()), None).await?;
|
||||
|
||||
if response.status() != StatusCode::OK {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
return Err(format!("attach policy to user failed: {status} {body}").into());
|
||||
}
|
||||
|
||||
Ok(())
|
||||
admin_attach_user_policy_via(AdminTransport::Signed, &env.url, &env.access_key, &env.secret_key, policy_name, username).await
|
||||
}
|
||||
|
||||
async fn admin_update_group_members(
|
||||
@@ -1941,6 +1912,21 @@ async fn site_replication_info(env: &RustFSTestEnvironment) -> Result<SiteReplic
|
||||
Ok(serde_json::from_slice(&response.bytes().await?)?)
|
||||
}
|
||||
|
||||
async fn site_replication_rotate_svc_acct(
|
||||
env: &RustFSTestEnvironment,
|
||||
) -> Result<ReplicateEditStatus, Box<dyn Error + Send + Sync>> {
|
||||
let url = format!("{}/rustfs/admin/v3/site-replication/rotate-svc-acct", env.url);
|
||||
let response = signed_request(http::Method::POST, &url, &env.access_key, &env.secret_key, None, None).await?;
|
||||
|
||||
if response.status() != StatusCode::OK {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
return Err(format!("site replication rotate-svc-acct failed: {status} {body}").into());
|
||||
}
|
||||
|
||||
Ok(serde_json::from_slice(&response.bytes().await?)?)
|
||||
}
|
||||
|
||||
async fn site_replication_resync_op(
|
||||
env: &RustFSTestEnvironment,
|
||||
operation: &str,
|
||||
@@ -6339,6 +6325,112 @@ async fn test_site_replication_remove_all_real_dual_node() -> Result<(), Box<dyn
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_site_replication_rotate_svc_acct_completes_and_replication_survives_real_dual_node()
|
||||
-> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
init_logging();
|
||||
|
||||
let mut source_env = RustFSTestEnvironment::new().await?;
|
||||
source_env
|
||||
.start_rustfs_server_with_env(vec![], LOOPBACK_REPLICATION_TARGET_ENV)
|
||||
.await?;
|
||||
|
||||
let mut target_env = RustFSTestEnvironment::new().await?;
|
||||
target_env
|
||||
.start_rustfs_server_without_cleanup_with_env(LOOPBACK_REPLICATION_TARGET_ENV)
|
||||
.await?;
|
||||
|
||||
let source_client = source_env.create_s3_client();
|
||||
let target_client = target_env.create_s3_client();
|
||||
let bucket = "site-repl-rotate-svc-acct";
|
||||
|
||||
let add_status = site_replication_add(
|
||||
&source_env,
|
||||
&[
|
||||
PeerSite {
|
||||
name: "source-site".to_string(),
|
||||
endpoint: source_env.url.clone(),
|
||||
access_key: source_env.access_key.clone(),
|
||||
secret_key: source_env.secret_key.clone(),
|
||||
..Default::default()
|
||||
},
|
||||
PeerSite {
|
||||
name: "target-site".to_string(),
|
||||
endpoint: target_env.url.clone(),
|
||||
access_key: target_env.access_key.clone(),
|
||||
secret_key: target_env.secret_key.clone(),
|
||||
..Default::default()
|
||||
},
|
||||
],
|
||||
)
|
||||
.await?;
|
||||
assert!(add_status.success, "unexpected site add result: {add_status:?}");
|
||||
|
||||
let _source_info = wait_for_site_replication_enabled(&source_env, 2).await?;
|
||||
let _target_info = wait_for_site_replication_enabled(&target_env, 2).await?;
|
||||
|
||||
source_client.create_bucket().bucket(bucket).send().await?;
|
||||
enable_bucket_versioning(&source_env, bucket).await?;
|
||||
wait_for_bucket_on_target(&target_client, bucket).await?;
|
||||
let baseline_payload = b"before rotation".to_vec();
|
||||
source_client
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key("before-rotate.txt")
|
||||
.body(ByteStream::from(baseline_payload.clone()))
|
||||
.send()
|
||||
.await?;
|
||||
let replicated_baseline = wait_for_object_on_target(&target_client, bucket, "before-rotate.txt").await?;
|
||||
assert_eq!(replicated_baseline, baseline_payload);
|
||||
|
||||
// A single rotation call must finish the whole hand-over. Before the fix
|
||||
// the join push could only sign with the freshly installed secret, every
|
||||
// peer rejected it, the rotation stayed pending forever, and both
|
||||
// replication directions were dead until an operator retried.
|
||||
let rotate_status = site_replication_rotate_svc_acct(&source_env).await?;
|
||||
assert!(rotate_status.success, "rotation did not complete in one call: {rotate_status:?}");
|
||||
|
||||
for env in [&source_env, &target_env] {
|
||||
let deadline = std::time::Instant::now() + Duration::from_secs(30);
|
||||
loop {
|
||||
let info = site_replication_info(env).await?;
|
||||
if info.enabled && info.pending_operation.is_none() {
|
||||
break;
|
||||
}
|
||||
if std::time::Instant::now() > deadline {
|
||||
return Err(format!("rotation left {} with a pending operation: {:?}", env.url, info.pending_operation).into());
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||
}
|
||||
}
|
||||
|
||||
// Replication must actually flow again in both directions with the
|
||||
// rotated service-account secret.
|
||||
let forward_payload = b"after rotation from source".to_vec();
|
||||
source_client
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key("after-rotate-forward.txt")
|
||||
.body(ByteStream::from(forward_payload.clone()))
|
||||
.send()
|
||||
.await?;
|
||||
let replicated_forward = wait_for_object_on_target(&target_client, bucket, "after-rotate-forward.txt").await?;
|
||||
assert_eq!(replicated_forward, forward_payload);
|
||||
|
||||
let reverse_payload = b"after rotation from target".to_vec();
|
||||
target_client
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key("after-rotate-reverse.txt")
|
||||
.body(ByteStream::from(reverse_payload.clone()))
|
||||
.send()
|
||||
.await?;
|
||||
let replicated_reverse = wait_for_object_on_target(&source_client, bucket, "after-rotate-reverse.txt").await?;
|
||||
assert_eq!(replicated_reverse, reverse_payload);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_site_replication_state_edit_fresh_and_stale_real_dual_node() -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
init_logging();
|
||||
|
||||
@@ -352,6 +352,28 @@ pub struct BucketTargetSys {
|
||||
heartbeat_started: OnceLock<()>,
|
||||
}
|
||||
|
||||
/// Build the bucket-target health-check HTTP client without panicking when
|
||||
/// the host has no system CA bundle (issue #6734).
|
||||
///
|
||||
/// `BucketTargetSys::get()` initializes lazily on the startup path (bucket
|
||||
/// metadata install calls it on the main thread), and `reqwest::Client::new()`
|
||||
/// panics when the TLS backend cannot load any system trust root — the state
|
||||
/// of a minimal container image. Fall back to a client with an explicit empty
|
||||
/// trust store: HTTP health checks keep working, and HTTPS targets fail closed
|
||||
/// at the TLS handshake with a clear certificate error instead of aborting
|
||||
/// the whole process at startup.
|
||||
fn build_health_check_client() -> HttpClient {
|
||||
HttpClient::builder().build().unwrap_or_else(|error| {
|
||||
warn!(
|
||||
"bucket target health-check HTTP client could not load system TLS roots ({error}); continuing with an empty trust store — HTTPS target health checks will fail until a CA bundle is installed"
|
||||
);
|
||||
HttpClient::builder()
|
||||
.tls_certs_only(std::iter::empty::<reqwest::Certificate>())
|
||||
.build()
|
||||
.expect("HTTP client construction must succeed with an explicit empty trust store")
|
||||
})
|
||||
}
|
||||
|
||||
impl BucketTargetSys {
|
||||
pub fn get() -> &'static Self {
|
||||
GLOBAL_BUCKET_TARGET_SYS.get_or_init(Self::new)
|
||||
@@ -364,7 +386,7 @@ impl BucketTargetSys {
|
||||
targets_map: Arc::new(RwLock::new(HashMap::new())),
|
||||
h_mutex: Arc::new(RwLock::new(HashMap::new())),
|
||||
target_h_mutex: Arc::new(RwLock::new(HashMap::new())),
|
||||
hc_client: Arc::new(HttpClient::new()),
|
||||
hc_client: Arc::new(build_health_check_client()),
|
||||
a_mutex: Arc::new(Mutex::new(HashMap::new())),
|
||||
arn_errs_map: Arc::new(RwLock::new(HashMap::new())),
|
||||
target_update_mutexes: Arc::new(Mutex::new(HashMap::new())),
|
||||
@@ -2490,6 +2512,18 @@ mod tests {
|
||||
use super::*;
|
||||
use rcgen::generate_simple_self_signed;
|
||||
|
||||
// The startup panic fix for hosts without a CA bundle (issue #6734) rests
|
||||
// on two properties: the health-check client constructor never panics, and
|
||||
// its degraded fallback — an explicit empty trust store — always builds.
|
||||
#[test]
|
||||
fn health_check_client_construction_never_panics() {
|
||||
let _ = build_health_check_client();
|
||||
HttpClient::builder()
|
||||
.tls_certs_only(std::iter::empty::<reqwest::Certificate>())
|
||||
.build()
|
||||
.expect("empty-trust-store client build must succeed without touching system roots");
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
struct RecordingHttpConnector {
|
||||
request_uris: Arc<std::sync::Mutex<Vec<String>>>,
|
||||
|
||||
@@ -20,14 +20,13 @@ mod durable_namespace;
|
||||
pub mod evaluator;
|
||||
pub mod manual_transition_job;
|
||||
mod metadata_boundary;
|
||||
pub(crate) use metadata_boundary::{LifecycleExpiryConfigs, get_expiry_configs};
|
||||
pub(crate) use metadata_boundary::{LifecycleExpiryConfigs, get_expiry_configs, get_lifecycle_config};
|
||||
mod object_handlers_common;
|
||||
mod object_lock_boundary;
|
||||
pub use self::core as lifecycle;
|
||||
mod replication_sink;
|
||||
pub mod rule;
|
||||
mod runtime_boundary;
|
||||
mod tagging_boundary;
|
||||
pub mod tier_delete_journal;
|
||||
pub mod tier_free_version_recovery;
|
||||
pub mod tier_last_day_stats;
|
||||
|
||||
@@ -1,37 +0,0 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
#[allow(
|
||||
dead_code,
|
||||
reason = "declared boundary surface for the ECStore replication split plan; no caller in this port (backlog#1823)"
|
||||
)]
|
||||
pub(crate) fn decode_tags_to_map(tags: &str) -> HashMap<String, String> {
|
||||
crate::bucket::tagging::decode_tags_to_map(tags)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::decode_tags_to_map;
|
||||
|
||||
#[test]
|
||||
fn decode_tags_to_map_preserves_bucket_tagging_parser_behavior() {
|
||||
let tags = decode_tags_to_map("env=prod&encoded=a%2Fb&=ignored");
|
||||
|
||||
assert_eq!(tags.get("env").map(String::as_str), Some("prod"));
|
||||
assert_eq!(tags.get("encoded").map(String::as_str), Some("a/b"));
|
||||
assert!(!tags.contains_key(""));
|
||||
}
|
||||
}
|
||||
@@ -27,7 +27,7 @@ use rustfs_protos::{
|
||||
ConnectionEvictionLogLevel, evict_failed_connection_with_log_level, models::PingBodyBuilder,
|
||||
proto_gen::node_service::node_service_client::NodeServiceClient,
|
||||
};
|
||||
use std::time::Duration;
|
||||
use std::{sync::OnceLock, time::Duration};
|
||||
use tokio::time::timeout;
|
||||
use tonic::Request;
|
||||
use tonic::service::interceptor::InterceptedService;
|
||||
@@ -44,11 +44,35 @@ pub struct RemoteClient {
|
||||
}
|
||||
|
||||
impl RemoteClient {
|
||||
const ONLINE_CHECK_RESOURCE: &'static str = "health-lock-online";
|
||||
|
||||
pub fn new(endpoint: String) -> Self {
|
||||
Self { addr: endpoint }
|
||||
}
|
||||
|
||||
fn ping_body() -> Bytes {
|
||||
static BODY: OnceLock<Bytes> = OnceLock::new();
|
||||
BODY.get_or_init(|| {
|
||||
let mut fbb = flatbuffers::FlatBufferBuilder::new();
|
||||
let payload = fbb.create_vector(b"health-check");
|
||||
let mut builder = PingBodyBuilder::new(&mut fbb);
|
||||
builder.add_payload(payload);
|
||||
let root = builder.finish();
|
||||
fbb.finish(root, None);
|
||||
Bytes::copy_from_slice(fbb.finished_data())
|
||||
})
|
||||
.clone()
|
||||
}
|
||||
|
||||
fn build_ping_request() -> PingRequest {
|
||||
PingRequest {
|
||||
version: 1,
|
||||
body: Self::ping_body(),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn build_fresh_ping_request_for_test() -> PingRequest {
|
||||
let mut fbb = flatbuffers::FlatBufferBuilder::new();
|
||||
let payload = fbb.create_vector(b"health-check");
|
||||
let mut builder = PingBodyBuilder::new(&mut fbb);
|
||||
@@ -164,6 +188,16 @@ impl RemoteClient {
|
||||
)
|
||||
}
|
||||
|
||||
fn online_check_timeout() -> Duration {
|
||||
Duration::from_millis(
|
||||
rustfs_utils::get_env_u64(
|
||||
rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS,
|
||||
rustfs_config::DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS,
|
||||
)
|
||||
.max(1),
|
||||
)
|
||||
}
|
||||
|
||||
async fn execute_rpc<T, F>(&self, op: &'static str, resource_summary: &str, future: F) -> std::result::Result<T, LockError>
|
||||
where
|
||||
F: std::future::Future<Output = std::result::Result<T, tonic::Status>>,
|
||||
@@ -547,24 +581,37 @@ impl LockClient for RemoteClient {
|
||||
}
|
||||
|
||||
async fn is_online(&self) -> bool {
|
||||
// Use Ping interface to test if remote service is online
|
||||
let mut client = match self.get_client().await {
|
||||
Ok(client) => client,
|
||||
Err(_) => {
|
||||
info!("remote client {} connection failed", self.addr);
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
let ping_req = Request::new(Self::build_ping_request());
|
||||
|
||||
match client.ping(ping_req).await {
|
||||
Ok(_) => {
|
||||
info!("remote client {} is online", self.addr);
|
||||
let online_timeout = Self::online_check_timeout();
|
||||
match timeout(online_timeout, async {
|
||||
let mut client = self.get_client().await?;
|
||||
let ping_req = Request::new(Self::build_ping_request());
|
||||
self.execute_rpc("ping", Self::ONLINE_CHECK_RESOURCE, client.ping(ping_req))
|
||||
.await?;
|
||||
Ok::<(), LockError>(())
|
||||
})
|
||||
.await
|
||||
{
|
||||
Ok(Ok(())) => {
|
||||
debug!(addr = %self.addr, timeout_ms = online_timeout.as_millis(), "remote lock client is online");
|
||||
true
|
||||
}
|
||||
Ok(Err(err)) => {
|
||||
debug!(
|
||||
addr = %self.addr,
|
||||
timeout_ms = online_timeout.as_millis(),
|
||||
error = %err,
|
||||
"remote lock client online check failed"
|
||||
);
|
||||
false
|
||||
}
|
||||
Err(_) => {
|
||||
info!("remote client {} ping failed", self.addr);
|
||||
let reason = format!("online check timed out after {:?}", online_timeout);
|
||||
warn!(
|
||||
addr = %self.addr,
|
||||
timeout_ms = online_timeout.as_millis(),
|
||||
"remote lock client online check timed out"
|
||||
);
|
||||
self.evict_connection("ping", &reason, Self::ONLINE_CHECK_RESOURCE).await;
|
||||
false
|
||||
}
|
||||
}
|
||||
@@ -651,6 +698,15 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cached_ping_request_matches_fresh_flatbuffer_payload() {
|
||||
let cached = RemoteClient::build_ping_request();
|
||||
let fresh = RemoteClient::build_fresh_ping_request_for_test();
|
||||
|
||||
assert_eq!(cached.version, fresh.version);
|
||||
assert_eq!(cached.body, fresh.body);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn test_remote_client_acquire_lock_uses_rpc_timeout_and_evicts_connection() {
|
||||
@@ -779,6 +835,48 @@ mod tests {
|
||||
accept_task.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn test_remote_client_is_online_uses_health_timeout_and_evicts_connection() {
|
||||
ensure_test_rpc_secret();
|
||||
let Some((addr, accept_task)) = spawn_hanging_listener().await else {
|
||||
return;
|
||||
};
|
||||
cache_lazy_channel(&addr).await;
|
||||
assert!(runtime_sources::test_node_channel_is_cached(&addr).await);
|
||||
|
||||
temp_env::async_with_vars(
|
||||
[
|
||||
(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("50")),
|
||||
(rustfs_config::ENV_OBJECT_LOCK_RPC_TIMEOUT_MS, Some("1000")),
|
||||
],
|
||||
async {
|
||||
let client = RemoteClient::new(addr.clone());
|
||||
let started_at = tokio::time::Instant::now();
|
||||
|
||||
let online = client.is_online().await;
|
||||
let elapsed = started_at.elapsed();
|
||||
|
||||
assert!(!online, "hanging remote lock peer must not be reported online");
|
||||
assert!(
|
||||
elapsed >= Duration::from_millis(40),
|
||||
"remote online check should honor configured health timeout, got {elapsed:?}"
|
||||
);
|
||||
assert!(
|
||||
elapsed < Duration::from_secs(1),
|
||||
"health timeout should keep readiness probes bounded, got {elapsed:?}"
|
||||
);
|
||||
assert!(
|
||||
!runtime_sources::test_node_channel_is_cached(&addr).await,
|
||||
"online-check timeout should evict cached connection"
|
||||
);
|
||||
},
|
||||
)
|
||||
.await;
|
||||
|
||||
accept_task.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn test_remote_client_refresh_tonic_error_evicts_connection() {
|
||||
@@ -906,4 +1004,21 @@ mod tests {
|
||||
assert_eq!(RemoteClient::rpc_timeout(), Duration::from_millis(1));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial]
|
||||
fn test_remote_client_online_timeout_honors_configured_deadline() {
|
||||
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, None::<&str>, || {
|
||||
assert_eq!(
|
||||
RemoteClient::online_check_timeout(),
|
||||
Duration::from_millis(rustfs_config::DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS)
|
||||
);
|
||||
});
|
||||
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("50"), || {
|
||||
assert_eq!(RemoteClient::online_check_timeout(), Duration::from_millis(50));
|
||||
});
|
||||
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("0"), || {
|
||||
assert_eq!(RemoteClient::online_check_timeout(), Duration::from_millis(1));
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -12,35 +12,28 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::target_defaults::{amqp_kvs, kafka_kvs, mysql_kvs, nats_kvs, postgres_kvs, pulsar_kvs, redis_kvs};
|
||||
use rustfs_config::audit::AUDIT_REDIS_DEFAULT_CHANNEL;
|
||||
use rustfs_config::server_config::{KV, KVS};
|
||||
use rustfs_config::{
|
||||
AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
|
||||
AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
|
||||
EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_SASL_ENABLE,
|
||||
KAFKA_SASL_MECHANISM, KAFKA_SASL_PASSWORD, KAFKA_SASL_USERNAME, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY,
|
||||
KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR,
|
||||
MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY,
|
||||
MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT,
|
||||
MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT,
|
||||
MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS,
|
||||
NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR,
|
||||
NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN,
|
||||
NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE,
|
||||
POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER,
|
||||
PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
|
||||
PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT,
|
||||
REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD,
|
||||
REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT,
|
||||
REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL,
|
||||
REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY,
|
||||
WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL,
|
||||
WEBHOOK_SKIP_TLS_VERIFY,
|
||||
COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY, EVENT_DEFAULT_DIR, EnableState, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD,
|
||||
MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY,
|
||||
MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, WEBHOOK_AUTH_TOKEN,
|
||||
WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT,
|
||||
WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, WEBHOOK_SKIP_TLS_VERIFY,
|
||||
};
|
||||
use std::sync::LazyLock;
|
||||
|
||||
#[allow(clippy::declare_interior_mutable_const)]
|
||||
/// Default KVS for audit webhook settings.
|
||||
///
|
||||
/// `WEBHOOK_BATCH_SIZE`/`WEBHOOK_MAX_RETRY`/`WEBHOOK_RETRY_INTERVAL`/`WEBHOOK_HTTP_TIMEOUT`
|
||||
/// exist here but not in [`crate::config::notify::DEFAULT_NOTIFY_WEBHOOK_KVS`]. This mirrors
|
||||
/// MinIO upstream: `internal/logger/config.go`'s `DefaultAuditWebhookKVS` carries the same
|
||||
/// four keys with the same defaults (`"1"`/`"0"`/`"3s"`/`"5s"`), while
|
||||
/// `internal/config/notify/parse.go`'s `DefaultWebhookKVS` (bucket event notifications) does
|
||||
/// not — the notify webhook delivery path never supported them. Not a copy/paste gap
|
||||
/// (backlog#2054).
|
||||
pub static DEFAULT_AUDIT_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
@@ -56,7 +49,7 @@ pub static DEFAULT_AUDIT_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KV {
|
||||
key: WEBHOOK_AUTH_TOKEN.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
hidden_if_empty: true, // Sensitive field; matches notify's webhook auth_token (backlog#2054)
|
||||
},
|
||||
KV {
|
||||
key: WEBHOOK_CLIENT_CERT.to_owned(),
|
||||
@@ -118,6 +111,15 @@ pub static DEFAULT_AUDIT_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
|
||||
#[allow(clippy::declare_interior_mutable_const)]
|
||||
/// Default KVS for audit MQTT settings.
|
||||
///
|
||||
/// `MQTT_QOS`/`MQTT_KEEP_ALIVE_INTERVAL`/`MQTT_RECONNECT_INTERVAL` default to a stronger
|
||||
/// delivery posture here (`"1"`/`"60s"`/`"5s"`) than
|
||||
/// [`crate::config::notify::DEFAULT_NOTIFY_MQTT_KVS`] (`"0"`/`"0s"`/`"0s"`, which matches
|
||||
/// MinIO's own `DefaultMQTTKVS` in `internal/config/notify/parse.go` byte-for-byte). MinIO has
|
||||
/// no MQTT audit target to compare against — audit-over-MQTT is a RustFS-original addition —
|
||||
/// so this divergence cannot be checked against upstream; it is intentional (audit favors
|
||||
/// at-least-once delivery and faster reconnect over notify's opt-in defaults), not a
|
||||
/// copy/paste gap (backlog#2054).
|
||||
pub static DEFAULT_AUDIT_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
@@ -208,542 +210,18 @@ pub static DEFAULT_AUDIT_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
])
|
||||
});
|
||||
|
||||
pub static DEFAULT_AUDIT_AMQP_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_URL.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_EXCHANGE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_ROUTING_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_MANDATORY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_PERSISTENT.to_owned(),
|
||||
value: EnableState::On.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
// The remaining targets declare the same defaults as notify, so both sides build them from
|
||||
// `target_defaults`. Redis and mysql pass in the single default that audit and notify disagree on.
|
||||
pub static DEFAULT_AUDIT_AMQP_KVS: LazyLock<KVS> = LazyLock::new(amqp_kvs);
|
||||
|
||||
pub static DEFAULT_AUDIT_NATS_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_ADDRESS.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_SUBJECT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TOKEN.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_CREDENTIALS_FILE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_REQUIRED.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_JETSTREAM_ENABLE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_JETSTREAM_STREAM_NAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_owned(),
|
||||
value: NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
pub static DEFAULT_AUDIT_NATS_KVS: LazyLock<KVS> = LazyLock::new(nats_kvs);
|
||||
|
||||
pub static DEFAULT_AUDIT_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_BROKER.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TOPIC.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_AUTH_TOKEN.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TLS_ALLOW_INSECURE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TLS_HOSTNAME_VERIFICATION.to_owned(),
|
||||
value: EnableState::On.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
pub static DEFAULT_AUDIT_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(pulsar_kvs);
|
||||
|
||||
pub static DEFAULT_AUDIT_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_URL.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_CHANNEL.to_owned(),
|
||||
value: AUDIT_REDIS_DEFAULT_CHANNEL.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_KEEP_ALIVE_INTERVAL.to_owned(),
|
||||
value: "15".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_MAX_RETRY_ATTEMPTS.to_owned(),
|
||||
value: "3".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_RECONNECT_RETRY_ATTEMPTS.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_MIN_RETRY_DELAY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_MAX_RETRY_DELAY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_CONNECTION_TIMEOUT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_RESPONSE_TIMEOUT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_PIPELINE_BUFFER_SIZE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_POLICY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_ALLOW_INSECURE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
pub static DEFAULT_AUDIT_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| redis_kvs(AUDIT_REDIS_DEFAULT_CHANNEL));
|
||||
|
||||
pub static DEFAULT_AUDIT_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_DSN_STRING.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TABLE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_FORMAT.to_owned(),
|
||||
value: "namespace".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_REQUIRED.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
pub static DEFAULT_AUDIT_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(postgres_kvs);
|
||||
|
||||
pub static DEFAULT_AUDIT_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_BROKERS.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TOPIC.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_ACKS.to_owned(),
|
||||
value: "1".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_ENABLE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_ENABLE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_MECHANISM.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
pub static DEFAULT_AUDIT_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(kafka_kvs);
|
||||
|
||||
pub static DEFAULT_AUDIT_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_DSN_STRING.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TABLE.to_owned(),
|
||||
value: "rustfs_audit_logs".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_FORMAT.to_owned(),
|
||||
value: "access".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_MAX_OPEN_CONNECTIONS.to_owned(),
|
||||
value: "2".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
pub static DEFAULT_AUDIT_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| mysql_kvs("rustfs_audit_logs"));
|
||||
|
||||
@@ -2793,12 +2793,13 @@ where
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{
|
||||
SERVER_CONFIG_LOCK, ServerConfigSnapshot, apply_dynamic_config_for_sub_sys_with, config_task_join_error,
|
||||
configs_semantically_equal, decode_server_config_blob, encode_server_config_blob, is_standard_object_server_config,
|
||||
lookup_configs, new_and_save_server_config, read_config, read_config_no_lock_preserve_empty_with_metadata,
|
||||
read_config_preserve_empty, read_config_with_metadata, read_config_without_migrate, read_server_config_snapshot,
|
||||
save_server_config, save_server_config_snapshot, save_server_config_snapshot_with_generation,
|
||||
server_config_transaction_lock_path, should_warn_ignored_scalar_section, storage_class_kvs_mut,
|
||||
SERVER_CONFIG_LOCK, ServerConfigSnapshot, apply_dynamic_config_for_sub_sys_with, build_scalar_config_object,
|
||||
config_task_join_error, configs_semantically_equal, decode_server_config_blob, encode_server_config_blob,
|
||||
heal_config_descriptor, is_standard_object_server_config, lookup_configs, new_and_save_server_config, read_config,
|
||||
read_config_no_lock_preserve_empty_with_metadata, read_config_preserve_empty, read_config_with_metadata,
|
||||
read_config_without_migrate, read_server_config_snapshot, save_server_config, save_server_config_snapshot,
|
||||
save_server_config_snapshot_with_generation, server_config_transaction_lock_path, should_warn_ignored_scalar_section,
|
||||
storage_class_kvs_mut,
|
||||
};
|
||||
use crate::config::{audit, heal, notify, oidc, scanner};
|
||||
use crate::disk::endpoint::Endpoint;
|
||||
@@ -3541,6 +3542,31 @@ mod tests {
|
||||
cfg
|
||||
}
|
||||
|
||||
/// `Config::new()` (and every decode built on it) reads the process-global
|
||||
/// `rustfs_config::server_config::DEFAULT_KVS` OnceLock at call time, and
|
||||
/// other tests in this binary register it via `crate::config::init()`
|
||||
/// mid-run. Equality assertions must therefore normalize both sides with
|
||||
/// one snapshot taken after both configs exist, never against a later
|
||||
/// `Config::new()`.
|
||||
fn default_kvs_snapshot() -> Option<&'static std::collections::HashMap<String, KVS>> {
|
||||
rustfs_config::server_config::DEFAULT_KVS.get()
|
||||
}
|
||||
|
||||
/// Fills the default sections `cfg` is missing from an explicit
|
||||
/// [`default_kvs_snapshot`], mirroring `Config::set_defaults`.
|
||||
fn filled_with_default_kvs(mut cfg: Config, snapshot: Option<&std::collections::HashMap<String, KVS>>) -> Config {
|
||||
if let Some(defaults) = snapshot {
|
||||
for (sub_sys, kvs) in defaults {
|
||||
cfg.0
|
||||
.entry(sub_sys.clone())
|
||||
.or_default()
|
||||
.entry(DEFAULT_DELIMITER.to_string())
|
||||
.or_insert_with(|| kvs.clone());
|
||||
}
|
||||
}
|
||||
cfg
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_external_scanner_config_decodes_with_defaults() {
|
||||
let cfg =
|
||||
@@ -3630,7 +3656,9 @@ mod tests {
|
||||
}"#;
|
||||
|
||||
let cfg = decode_server_config_blob(seed).expect("root heal null should mean no persisted override");
|
||||
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
|
||||
// The heal section may hold registered defaults, so assert on the
|
||||
// semantic diff instead of the section's presence.
|
||||
assert!(build_scalar_config_object(&cfg, heal_config_descriptor()).is_empty());
|
||||
assert!(!is_standard_object_server_config(seed));
|
||||
|
||||
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("legacy seed should canonicalize on an authorized save");
|
||||
@@ -3665,7 +3693,12 @@ mod tests {
|
||||
let input = format!(r#"{{"version":"33","storageclass":{{"standard":"","rrs":""}},{section}}}"#);
|
||||
let cfg = decode_server_config_blob(input.as_bytes())
|
||||
.unwrap_or_else(|err| panic!("legacy scalar section {section} should be ignored, got: {err}"));
|
||||
assert_eq!(cfg, base, "ignored section {section} must contribute no overrides");
|
||||
let snapshot = default_kvs_snapshot();
|
||||
assert_eq!(
|
||||
filled_with_default_kvs(cfg.clone(), snapshot),
|
||||
filled_with_default_kvs(base.clone(), snapshot),
|
||||
"ignored section {section} must contribute no overrides"
|
||||
);
|
||||
assert!(
|
||||
!is_standard_object_server_config(input.as_bytes()),
|
||||
"seed with {section} must not count as standard so a save rewrites it"
|
||||
@@ -3743,12 +3776,19 @@ mod tests {
|
||||
fn valid_heal_object_and_kvs_array_shapes_remain_accepted() {
|
||||
let empty_object = br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":{}}"#;
|
||||
let cfg = decode_server_config_blob(empty_object).expect("empty heal object should decode as no override");
|
||||
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
|
||||
// The heal section may hold registered defaults, so assert on the
|
||||
// semantic diff instead of the section's presence.
|
||||
assert!(build_scalar_config_object(&cfg, heal_config_descriptor()).is_empty());
|
||||
|
||||
let kvs_array =
|
||||
br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":[{"key":"bitrot_cycle","value":"off"}]}"#;
|
||||
let cfg = decode_server_config_blob(kvs_array).expect("heal KVS array should decode");
|
||||
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_some());
|
||||
assert_eq!(
|
||||
build_scalar_config_object(&cfg, heal_config_descriptor())
|
||||
.get(HEAL_BITROT_CYCLE)
|
||||
.and_then(Value::as_str),
|
||||
Some("off")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -4900,8 +4940,12 @@ mod tests {
|
||||
fn test_fallback_returns_default_config_when_recovery_enabled() {
|
||||
let cfg = fallback_server_config_after_corruption(corrupt_config_error(), "config/config.json", true)
|
||||
.expect("recovery enabled must fall back to the default config");
|
||||
let snapshot = default_kvs_snapshot();
|
||||
assert!(
|
||||
configs_semantically_equal(&cfg, &Config::new()),
|
||||
configs_semantically_equal(
|
||||
&filled_with_default_kvs(cfg, snapshot),
|
||||
&filled_with_default_kvs(Config(std::collections::HashMap::new()), snapshot)
|
||||
),
|
||||
"fallback config should be the default server config"
|
||||
);
|
||||
}
|
||||
@@ -5518,8 +5562,12 @@ mod tests {
|
||||
.expect("unrecoverable corruption should fall back to the default config");
|
||||
|
||||
assert_eq!(store.heal_calls.load(Ordering::SeqCst), 1, "heal should be attempted before falling back");
|
||||
let snapshot = default_kvs_snapshot();
|
||||
assert!(
|
||||
configs_semantically_equal(&cfg, &Config::new()),
|
||||
configs_semantically_equal(
|
||||
&filled_with_default_kvs(cfg, snapshot),
|
||||
&filled_with_default_kvs(Config(std::collections::HashMap::new()), snapshot)
|
||||
),
|
||||
"fallback config should be the default server config"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -21,6 +21,7 @@ mod notify;
|
||||
mod oidc;
|
||||
mod scanner;
|
||||
pub mod storageclass;
|
||||
mod target_defaults;
|
||||
|
||||
use crate::error::Result;
|
||||
use crate::store::ECStore;
|
||||
|
||||
@@ -12,34 +12,26 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::target_defaults::{amqp_kvs, kafka_kvs, mysql_kvs, nats_kvs, postgres_kvs, pulsar_kvs, redis_kvs};
|
||||
use rustfs_config::notify::NOTIFY_REDIS_DEFAULT_CHANNEL;
|
||||
use rustfs_config::server_config::{KV, KVS};
|
||||
use rustfs_config::{
|
||||
AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
|
||||
AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
|
||||
EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_SASL_ENABLE,
|
||||
KAFKA_SASL_MECHANISM, KAFKA_SASL_PASSWORD, KAFKA_SASL_USERNAME, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY,
|
||||
KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR,
|
||||
MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY,
|
||||
MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT,
|
||||
MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT,
|
||||
MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS,
|
||||
NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR,
|
||||
NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN,
|
||||
NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE,
|
||||
POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER,
|
||||
PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
|
||||
PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT,
|
||||
REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD,
|
||||
REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT,
|
||||
REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL,
|
||||
REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT,
|
||||
WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY,
|
||||
COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY, EVENT_DEFAULT_DIR, EnableState, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD,
|
||||
MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY,
|
||||
MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, WEBHOOK_AUTH_TOKEN,
|
||||
WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT,
|
||||
WEBHOOK_SKIP_TLS_VERIFY,
|
||||
};
|
||||
use std::sync::LazyLock;
|
||||
|
||||
/// The default configuration collection of webhooks,
|
||||
/// Initialized only once during the program life cycle, enabling high-performance lazy loading.
|
||||
///
|
||||
/// This table has no `batch_size`/`max_retry`/`retry_interval`/`http_timeout` keys, unlike
|
||||
/// [`crate::config::audit::DEFAULT_AUDIT_WEBHOOK_KVS`] — matching MinIO upstream, whose
|
||||
/// `internal/config/notify/parse.go` `DefaultWebhookKVS` (bucket event notifications) also
|
||||
/// omits them while `internal/logger/config.go`'s `DefaultAuditWebhookKVS` carries them.
|
||||
/// Intentional, not a copy/paste gap (backlog#2054).
|
||||
pub static DEFAULT_NOTIFY_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
@@ -97,6 +89,12 @@ pub static DEFAULT_NOTIFY_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
});
|
||||
|
||||
/// MQTT's default configuration collection
|
||||
///
|
||||
/// `MQTT_QOS`/`MQTT_KEEP_ALIVE_INTERVAL`/`MQTT_RECONNECT_INTERVAL` default to `"0"`/`"0s"`/`"0s"`
|
||||
/// here, matching MinIO's `DefaultMQTTKVS` in `internal/config/notify/parse.go`
|
||||
/// byte-for-byte — this table is a faithful port. [`crate::config::audit::DEFAULT_AUDIT_MQTT_KVS`]
|
||||
/// uses stronger, RustFS-original defaults instead (MinIO has no MQTT audit target to compare
|
||||
/// against); that divergence is intentional, not a copy/paste gap (backlog#2054).
|
||||
pub static DEFAULT_NOTIFY_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
@@ -188,543 +186,17 @@ pub static DEFAULT_NOTIFY_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
])
|
||||
});
|
||||
|
||||
pub static DEFAULT_NOTIFY_AMQP_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_URL.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_EXCHANGE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_ROUTING_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_MANDATORY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_PERSISTENT.to_owned(),
|
||||
value: EnableState::On.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
pub static DEFAULT_NOTIFY_AMQP_KVS: LazyLock<KVS> = LazyLock::new(amqp_kvs);
|
||||
|
||||
pub static DEFAULT_NOTIFY_NATS_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_ADDRESS.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_SUBJECT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TOKEN.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_CREDENTIALS_FILE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_REQUIRED.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_JETSTREAM_ENABLE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_JETSTREAM_STREAM_NAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_owned(),
|
||||
value: NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
pub static DEFAULT_NOTIFY_NATS_KVS: LazyLock<KVS> = LazyLock::new(nats_kvs);
|
||||
|
||||
pub static DEFAULT_NOTIFY_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_BROKER.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TOPIC.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_AUTH_TOKEN.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TLS_ALLOW_INSECURE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TLS_HOSTNAME_VERIFICATION.to_owned(),
|
||||
value: EnableState::On.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
pub static DEFAULT_NOTIFY_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(pulsar_kvs);
|
||||
|
||||
pub static DEFAULT_NOTIFY_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_URL.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_CHANNEL.to_owned(),
|
||||
value: NOTIFY_REDIS_DEFAULT_CHANNEL.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_KEEP_ALIVE_INTERVAL.to_owned(),
|
||||
value: "15".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_MAX_RETRY_ATTEMPTS.to_owned(),
|
||||
value: "3".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_RECONNECT_RETRY_ATTEMPTS.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_MIN_RETRY_DELAY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_MAX_RETRY_DELAY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_CONNECTION_TIMEOUT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_RESPONSE_TIMEOUT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_PIPELINE_BUFFER_SIZE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_POLICY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_ALLOW_INSECURE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
pub static DEFAULT_NOTIFY_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| redis_kvs(NOTIFY_REDIS_DEFAULT_CHANNEL));
|
||||
|
||||
pub static DEFAULT_NOTIFY_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_DSN_STRING.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TABLE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_FORMAT.to_owned(),
|
||||
value: "namespace".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_REQUIRED.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
pub static DEFAULT_NOTIFY_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(postgres_kvs);
|
||||
|
||||
pub static DEFAULT_NOTIFY_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_BROKERS.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TOPIC.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_ACKS.to_owned(),
|
||||
value: "1".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_ENABLE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_ENABLE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_MECHANISM.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
pub static DEFAULT_NOTIFY_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(kafka_kvs);
|
||||
|
||||
/// MySQL notification target default configuration
|
||||
pub static DEFAULT_NOTIFY_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_DSN_STRING.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TABLE.to_owned(),
|
||||
value: "rustfs_events".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_FORMAT.to_owned(),
|
||||
value: "access".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_MAX_OPEN_CONNECTIONS.to_owned(),
|
||||
value: "2".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
pub static DEFAULT_NOTIFY_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| mysql_kvs("rustfs_events"));
|
||||
|
||||
@@ -246,16 +246,7 @@ impl Config {
|
||||
}
|
||||
|
||||
let shard_size = shard_size as usize;
|
||||
// Keep the historical two-data-shard object budget while preventing
|
||||
// wider EC layouts from multiplying the maximum inline object size.
|
||||
// Use div_ceil to match the shard_file_size calculation (which also uses
|
||||
// div_ceil), avoiding a 1-byte rounding discrepancy that prevents inline
|
||||
// for objects right at the threshold.
|
||||
let inline_block = if self.initialized && self.inline_block_explicit {
|
||||
self.inline_block
|
||||
} else {
|
||||
DEFAULT_INLINE_OBJECT_BUDGET.div_ceil(data_shards).min(DEFAULT_INLINE_BLOCK)
|
||||
};
|
||||
let inline_block = self.effective_inline_block(data_shards);
|
||||
|
||||
if versioned {
|
||||
shard_size <= inline_block / 8
|
||||
@@ -264,6 +255,27 @@ impl Config {
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns the per-shard inline budget used by both write admission and
|
||||
/// legacy read fallback.
|
||||
///
|
||||
/// The default budget is scaled by the number of data shards so a wider EC
|
||||
/// layout does not silently increase the maximum inline object size. An
|
||||
/// explicitly configured `inline_block` remains a fixed per-shard limit for
|
||||
/// compatibility with deployments that opted into the historical policy.
|
||||
pub(crate) fn effective_inline_block(&self, data_shards: usize) -> usize {
|
||||
if data_shards == 0 {
|
||||
return 0;
|
||||
}
|
||||
|
||||
if self.initialized && self.inline_block_explicit {
|
||||
self.inline_block
|
||||
} else {
|
||||
// Keep the historical two-data-shard object budget while preventing
|
||||
// wider EC layouts from multiplying the maximum inline object size.
|
||||
DEFAULT_INLINE_OBJECT_BUDGET.div_ceil(data_shards).min(DEFAULT_INLINE_BLOCK)
|
||||
}
|
||||
}
|
||||
|
||||
pub fn inline_block(&self) -> usize {
|
||||
if !self.initialized {
|
||||
DEFAULT_INLINE_BLOCK
|
||||
@@ -602,6 +614,51 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn should_inline_keeps_ec8_and_ec12_object_boundaries_consistent() {
|
||||
let config = Config::default();
|
||||
let object_sizes = [128 * 1024, 256 * 1024, 512 * 1024, 1024 * 1024, 4 * 1024 * 1024];
|
||||
|
||||
for (data_shards, parity_shards) in [(8, 4), (12, 4)] {
|
||||
let erasure = crate::erasure::coding::Erasure::new(data_shards, parity_shards, 1024 * 1024);
|
||||
let mut previous = true;
|
||||
for object_size in object_sizes {
|
||||
let shard_size = erasure.shard_file_size(object_size);
|
||||
let inline = config.should_inline(shard_size, data_shards, false);
|
||||
|
||||
// The effective policy is monotonic across object sizes. This
|
||||
// table covers the boundaries that previously exposed the
|
||||
// fixed-shard read-ahead mismatch, including the 1 MiB case.
|
||||
assert!(!inline || previous, "inline decision must not re-enable at {object_size} bytes");
|
||||
previous = inline;
|
||||
}
|
||||
|
||||
assert!(
|
||||
!config.should_inline(erasure.shard_file_size(1024 * 1024), data_shards, false),
|
||||
"1 MiB must use the non-inline path for EC{data_shards}+{parity_shards}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn effective_inline_block_scales_default_budget_and_preserves_explicit_limit() {
|
||||
let config = Config::default();
|
||||
assert_eq!(config.effective_inline_block(8), 32 * 1024);
|
||||
assert_eq!(config.effective_inline_block(12), 21_846);
|
||||
assert_eq!(config.effective_inline_block(0), 0);
|
||||
|
||||
let explicit = lookup_config_for_pools_with_env(
|
||||
&KVS::new(),
|
||||
&[12],
|
||||
StorageClassEnvOverrides {
|
||||
inline_block: Some("128KiB".to_string()),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.expect("explicit inline block should resolve");
|
||||
assert_eq!(explicit.effective_inline_block(12), 128 * 1024);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn explicit_inline_block_preserves_fixed_per_shard_rollback() {
|
||||
let overrides = StorageClassEnvOverrides {
|
||||
|
||||
@@ -0,0 +1,414 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Shared default KVS tables for delivery targets that audit and notify declare identically.
|
||||
//!
|
||||
//! The audit and notify subsystems register one default KVS per delivery target. For amqp, nats,
|
||||
//! pulsar, postgres and kafka both sides declare byte-identical tables; for redis and mysql they
|
||||
//! differ only in a single default literal, which the caller passes in.
|
||||
//!
|
||||
//! Webhook and mqtt are deliberately absent: audit's webhook table carries extra batching/retry
|
||||
//! keys and both tables disagree on key order and on several defaults (mqtt qos, keep-alive and
|
||||
//! reconnect intervals), so they are real behavioral forks, not duplication.
|
||||
//!
|
||||
//! Key order is part of the contract: it drives the order admin config output lists the keys in,
|
||||
//! so every constructor reproduces the existing order exactly.
|
||||
|
||||
use rustfs_config::server_config::{KV, KVS};
|
||||
use rustfs_config::{
|
||||
AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
|
||||
AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
|
||||
EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_SASL_ENABLE,
|
||||
KAFKA_SASL_MECHANISM, KAFKA_SASL_PASSWORD, KAFKA_SASL_USERNAME, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY,
|
||||
KAFKA_TLS_ENABLE, KAFKA_TOPIC, MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR,
|
||||
MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS,
|
||||
NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS, NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE,
|
||||
NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT,
|
||||
NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR,
|
||||
POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY,
|
||||
POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT,
|
||||
PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL,
|
||||
REDIS_CONNECTION_TIMEOUT, REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY,
|
||||
REDIS_PASSWORD, REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS,
|
||||
REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY,
|
||||
REDIS_TLS_POLICY, REDIS_URL, REDIS_USERNAME,
|
||||
};
|
||||
|
||||
/// Builds one default entry. `hidden_if_empty` marks values the admin API elides when unset.
|
||||
fn kv(key: &str, value: impl Into<String>, hidden_if_empty: bool) -> KV {
|
||||
KV {
|
||||
key: key.to_owned(),
|
||||
value: value.into(),
|
||||
hidden_if_empty,
|
||||
}
|
||||
}
|
||||
|
||||
/// Default KVS for the amqp delivery target.
|
||||
// Unused until the audit/notify tables are migrated onto these constructors.
|
||||
#[allow(dead_code)]
|
||||
pub fn amqp_kvs() -> KVS {
|
||||
KVS(vec![
|
||||
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
|
||||
kv(AMQP_URL, "", false),
|
||||
kv(AMQP_EXCHANGE, "", false),
|
||||
kv(AMQP_ROUTING_KEY, "", false),
|
||||
kv(AMQP_MANDATORY, EnableState::Off.to_string(), false),
|
||||
kv(AMQP_PERSISTENT, EnableState::On.to_string(), false),
|
||||
kv(AMQP_USERNAME, "", false),
|
||||
kv(AMQP_PASSWORD, "", true),
|
||||
kv(AMQP_TLS_CA, "", true),
|
||||
kv(AMQP_TLS_CLIENT_CERT, "", true),
|
||||
kv(AMQP_TLS_CLIENT_KEY, "", true),
|
||||
kv(AMQP_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
|
||||
kv(AMQP_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
|
||||
kv(COMMENT_KEY, "", false),
|
||||
])
|
||||
}
|
||||
|
||||
/// Default KVS for the nats delivery target.
|
||||
// Unused until the audit/notify tables are migrated onto these constructors.
|
||||
#[allow(dead_code)]
|
||||
pub fn nats_kvs() -> KVS {
|
||||
KVS(vec![
|
||||
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
|
||||
kv(NATS_ADDRESS, "", false),
|
||||
kv(NATS_SUBJECT, "", false),
|
||||
kv(NATS_USERNAME, "", false),
|
||||
kv(NATS_PASSWORD, "", true),
|
||||
kv(NATS_TOKEN, "", true),
|
||||
kv(NATS_CREDENTIALS_FILE, "", true),
|
||||
kv(NATS_TLS_CA, "", true),
|
||||
kv(NATS_TLS_CLIENT_CERT, "", true),
|
||||
kv(NATS_TLS_CLIENT_KEY, "", true),
|
||||
kv(NATS_TLS_REQUIRED, EnableState::Off.to_string(), false),
|
||||
kv(NATS_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
|
||||
kv(NATS_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
|
||||
kv(NATS_JETSTREAM_ENABLE, EnableState::Off.to_string(), false),
|
||||
kv(NATS_JETSTREAM_STREAM_NAME, "", false),
|
||||
kv(
|
||||
NATS_JETSTREAM_ACK_TIMEOUT_SECS,
|
||||
NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(),
|
||||
false,
|
||||
),
|
||||
kv(COMMENT_KEY, "", false),
|
||||
])
|
||||
}
|
||||
|
||||
/// Default KVS for the pulsar delivery target.
|
||||
// Unused until the audit/notify tables are migrated onto these constructors.
|
||||
#[allow(dead_code)]
|
||||
pub fn pulsar_kvs() -> KVS {
|
||||
KVS(vec![
|
||||
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
|
||||
kv(PULSAR_BROKER, "", false),
|
||||
kv(PULSAR_TOPIC, "", false),
|
||||
kv(PULSAR_AUTH_TOKEN, "", true),
|
||||
kv(PULSAR_USERNAME, "", false),
|
||||
kv(PULSAR_PASSWORD, "", true),
|
||||
kv(PULSAR_TLS_CA, "", true),
|
||||
kv(PULSAR_TLS_ALLOW_INSECURE, EnableState::Off.to_string(), false),
|
||||
kv(PULSAR_TLS_HOSTNAME_VERIFICATION, EnableState::On.to_string(), false),
|
||||
kv(PULSAR_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
|
||||
kv(PULSAR_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
|
||||
kv(COMMENT_KEY, "", false),
|
||||
])
|
||||
}
|
||||
|
||||
/// Default KVS for the postgres delivery target.
|
||||
// Unused until the audit/notify tables are migrated onto these constructors.
|
||||
#[allow(dead_code)]
|
||||
pub fn postgres_kvs() -> KVS {
|
||||
KVS(vec![
|
||||
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
|
||||
kv(POSTGRES_DSN_STRING, "", true),
|
||||
kv(POSTGRES_TABLE, "", false),
|
||||
kv(POSTGRES_FORMAT, "namespace", false),
|
||||
kv(POSTGRES_TLS_REQUIRED, EnableState::Off.to_string(), false),
|
||||
kv(POSTGRES_TLS_CA, "", true),
|
||||
kv(POSTGRES_TLS_CLIENT_CERT, "", true),
|
||||
kv(POSTGRES_TLS_CLIENT_KEY, "", true),
|
||||
kv(POSTGRES_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
|
||||
kv(POSTGRES_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
|
||||
kv(COMMENT_KEY, "", false),
|
||||
])
|
||||
}
|
||||
|
||||
/// Default KVS for the kafka delivery target.
|
||||
// Unused until the audit/notify tables are migrated onto these constructors.
|
||||
#[allow(dead_code)]
|
||||
pub fn kafka_kvs() -> KVS {
|
||||
KVS(vec![
|
||||
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
|
||||
kv(KAFKA_BROKERS, "", false),
|
||||
kv(KAFKA_TOPIC, "", false),
|
||||
kv(KAFKA_ACKS, "1", false),
|
||||
kv(KAFKA_TLS_ENABLE, EnableState::Off.to_string(), false),
|
||||
kv(KAFKA_TLS_CA, "", true),
|
||||
kv(KAFKA_TLS_CLIENT_CERT, "", true),
|
||||
kv(KAFKA_TLS_CLIENT_KEY, "", true),
|
||||
kv(KAFKA_SASL_ENABLE, EnableState::Off.to_string(), false),
|
||||
kv(KAFKA_SASL_MECHANISM, "", false),
|
||||
kv(KAFKA_SASL_USERNAME, "", false),
|
||||
kv(KAFKA_SASL_PASSWORD, "", true),
|
||||
kv(KAFKA_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
|
||||
kv(KAFKA_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
|
||||
kv(COMMENT_KEY, "", false),
|
||||
])
|
||||
}
|
||||
|
||||
/// Default KVS for the redis delivery target. `channel` is the subsystem's default pub/sub channel,
|
||||
/// which is the only value audit and notify disagree on.
|
||||
// Unused until the audit/notify tables are migrated onto these constructors.
|
||||
#[allow(dead_code)]
|
||||
pub fn redis_kvs(channel: &str) -> KVS {
|
||||
KVS(vec![
|
||||
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
|
||||
kv(REDIS_URL, "", false),
|
||||
kv(REDIS_CHANNEL, channel, false),
|
||||
kv(REDIS_USERNAME, "", false),
|
||||
kv(REDIS_PASSWORD, "", true),
|
||||
kv(REDIS_KEEP_ALIVE_INTERVAL, "15", false),
|
||||
kv(REDIS_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
|
||||
kv(REDIS_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
|
||||
kv(REDIS_MAX_RETRY_ATTEMPTS, "3", false),
|
||||
kv(REDIS_RECONNECT_RETRY_ATTEMPTS, "", false),
|
||||
kv(REDIS_MIN_RETRY_DELAY, "", false),
|
||||
kv(REDIS_MAX_RETRY_DELAY, "", false),
|
||||
kv(REDIS_CONNECTION_TIMEOUT, "", false),
|
||||
kv(REDIS_RESPONSE_TIMEOUT, "", false),
|
||||
kv(REDIS_PIPELINE_BUFFER_SIZE, "", false),
|
||||
kv(REDIS_TLS_POLICY, "", true),
|
||||
kv(REDIS_TLS_CA, "", true),
|
||||
kv(REDIS_TLS_CLIENT_CERT, "", true),
|
||||
kv(REDIS_TLS_CLIENT_KEY, "", true),
|
||||
kv(REDIS_TLS_ALLOW_INSECURE, EnableState::Off.to_string(), false),
|
||||
kv(COMMENT_KEY, "", false),
|
||||
])
|
||||
}
|
||||
|
||||
/// Default KVS for the mysql delivery target. `table` is the subsystem's default destination table,
|
||||
/// which is the only value audit and notify disagree on.
|
||||
// Unused until the audit/notify tables are migrated onto these constructors.
|
||||
#[allow(dead_code)]
|
||||
pub fn mysql_kvs(table: &str) -> KVS {
|
||||
KVS(vec![
|
||||
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
|
||||
kv(MYSQL_DSN_STRING, "", true),
|
||||
kv(MYSQL_TABLE, table, false),
|
||||
kv(MYSQL_FORMAT, "access", false),
|
||||
kv(MYSQL_TLS_CA, "", true),
|
||||
kv(MYSQL_TLS_CLIENT_CERT, "", true),
|
||||
kv(MYSQL_TLS_CLIENT_KEY, "", true),
|
||||
kv(MYSQL_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
|
||||
kv(MYSQL_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
|
||||
kv(MYSQL_MAX_OPEN_CONNECTIONS, "2", false),
|
||||
kv(COMMENT_KEY, "", false),
|
||||
])
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Expected values are spelled out as literals on purpose: they mirror the tables currently
|
||||
/// declared in `audit.rs` and `notify.rs`, so a drift in key order or in any default breaks
|
||||
/// the test instead of silently changing admin config output.
|
||||
fn assert_table(actual: &KVS, expected: &[(&str, &str, bool)]) {
|
||||
let actual: Vec<(&str, &str, bool)> = actual
|
||||
.0
|
||||
.iter()
|
||||
.map(|kv| (kv.key.as_str(), kv.value.as_str(), kv.hidden_if_empty))
|
||||
.collect();
|
||||
assert_eq!(actual, expected);
|
||||
}
|
||||
|
||||
const QUEUE_DIR: &str = "/opt/rustfs/events";
|
||||
const QUEUE_LIMIT: &str = "100000";
|
||||
|
||||
#[test]
|
||||
fn amqp_table_matches_audit_and_notify() {
|
||||
assert_table(
|
||||
&amqp_kvs(),
|
||||
&[
|
||||
("enable", "off", false),
|
||||
("url", "", false),
|
||||
("exchange", "", false),
|
||||
("routing_key", "", false),
|
||||
("mandatory", "off", false),
|
||||
("persistent", "on", false),
|
||||
("username", "", false),
|
||||
("password", "", true),
|
||||
("tls_ca", "", true),
|
||||
("tls_client_cert", "", true),
|
||||
("tls_client_key", "", true),
|
||||
("queue_dir", QUEUE_DIR, false),
|
||||
("queue_limit", QUEUE_LIMIT, false),
|
||||
("comment", "", false),
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nats_table_matches_audit_and_notify() {
|
||||
assert_table(
|
||||
&nats_kvs(),
|
||||
&[
|
||||
("enable", "off", false),
|
||||
("address", "", false),
|
||||
("subject", "", false),
|
||||
("username", "", false),
|
||||
("password", "", true),
|
||||
("token", "", true),
|
||||
("credentials_file", "", true),
|
||||
("tls_ca", "", true),
|
||||
("tls_client_cert", "", true),
|
||||
("tls_client_key", "", true),
|
||||
("tls_required", "off", false),
|
||||
("queue_dir", QUEUE_DIR, false),
|
||||
("queue_limit", QUEUE_LIMIT, false),
|
||||
("jetstream_enable", "off", false),
|
||||
("jetstream_stream_name", "", false),
|
||||
("jetstream_ack_timeout_secs", "30", false),
|
||||
("comment", "", false),
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pulsar_table_matches_audit_and_notify() {
|
||||
assert_table(
|
||||
&pulsar_kvs(),
|
||||
&[
|
||||
("enable", "off", false),
|
||||
("broker", "", false),
|
||||
("topic", "", false),
|
||||
("auth_token", "", true),
|
||||
("username", "", false),
|
||||
("password", "", true),
|
||||
("tls_ca", "", true),
|
||||
("tls_allow_insecure", "off", false),
|
||||
("tls_hostname_verification", "on", false),
|
||||
("queue_dir", QUEUE_DIR, false),
|
||||
("queue_limit", QUEUE_LIMIT, false),
|
||||
("comment", "", false),
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn postgres_table_matches_audit_and_notify() {
|
||||
assert_table(
|
||||
&postgres_kvs(),
|
||||
&[
|
||||
("enable", "off", false),
|
||||
("dsn_string", "", true),
|
||||
("table", "", false),
|
||||
("format", "namespace", false),
|
||||
("tls_required", "off", false),
|
||||
("tls_ca", "", true),
|
||||
("tls_client_cert", "", true),
|
||||
("tls_client_key", "", true),
|
||||
("queue_dir", QUEUE_DIR, false),
|
||||
("queue_limit", QUEUE_LIMIT, false),
|
||||
("comment", "", false),
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn kafka_table_matches_audit_and_notify() {
|
||||
assert_table(
|
||||
&kafka_kvs(),
|
||||
&[
|
||||
("enable", "off", false),
|
||||
("brokers", "", false),
|
||||
("topic", "", false),
|
||||
("acks", "1", false),
|
||||
("tls_enable", "off", false),
|
||||
("tls_ca", "", true),
|
||||
("tls_client_cert", "", true),
|
||||
("tls_client_key", "", true),
|
||||
("sasl_enable", "off", false),
|
||||
("sasl_mechanism", "", false),
|
||||
("sasl_username", "", false),
|
||||
("sasl_password", "", true),
|
||||
("queue_dir", QUEUE_DIR, false),
|
||||
("queue_limit", QUEUE_LIMIT, false),
|
||||
("comment", "", false),
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
fn expected_redis(channel: &str) -> Vec<(&str, &str, bool)> {
|
||||
vec![
|
||||
("enable", "off", false),
|
||||
("url", "", false),
|
||||
("channel", channel, false),
|
||||
("username", "", false),
|
||||
("password", "", true),
|
||||
("keep_alive_interval", "15", false),
|
||||
("queue_dir", QUEUE_DIR, false),
|
||||
("queue_limit", QUEUE_LIMIT, false),
|
||||
("max_retry_attempts", "3", false),
|
||||
("reconnect_retry_attempts", "", false),
|
||||
("min_retry_delay", "", false),
|
||||
("max_retry_delay", "", false),
|
||||
("connection_timeout", "", false),
|
||||
("response_timeout", "", false),
|
||||
("pipeline_buffer_size", "", false),
|
||||
("tls_policy", "", true),
|
||||
("tls_ca", "", true),
|
||||
("tls_client_cert", "", true),
|
||||
("tls_client_key", "", true),
|
||||
("tls_allow_insecure", "off", false),
|
||||
("comment", "", false),
|
||||
]
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn redis_table_matches_audit() {
|
||||
assert_table(&redis_kvs("rustfs_audit_channel"), &expected_redis("rustfs_audit_channel"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn redis_table_matches_notify() {
|
||||
assert_table(&redis_kvs("rustfs_notify_channel"), &expected_redis("rustfs_notify_channel"));
|
||||
}
|
||||
|
||||
fn expected_mysql(table: &str) -> Vec<(&str, &str, bool)> {
|
||||
vec![
|
||||
("enable", "off", false),
|
||||
("dsn_string", "", true),
|
||||
("table", table, false),
|
||||
("format", "access", false),
|
||||
("tls_ca", "", true),
|
||||
("tls_client_cert", "", true),
|
||||
("tls_client_key", "", true),
|
||||
("queue_dir", QUEUE_DIR, false),
|
||||
("queue_limit", QUEUE_LIMIT, false),
|
||||
("max_open_connections", "2", false),
|
||||
("comment", "", false),
|
||||
]
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mysql_table_matches_audit() {
|
||||
assert_table(&mysql_kvs("rustfs_audit_logs"), &expected_mysql("rustfs_audit_logs"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mysql_table_matches_notify() {
|
||||
assert_table(&mysql_kvs("rustfs_events"), &expected_mysql("rustfs_events"));
|
||||
}
|
||||
}
|
||||
@@ -96,6 +96,8 @@ const LOG_SUBSYSTEM_POOLS: &str = "pools";
|
||||
const EVENT_DECOMMISSION_STATE: &str = "decommission_state";
|
||||
const EVENT_DECOMMISSION_BUCKET: &str = "decommission_bucket";
|
||||
const EVENT_DECOMMISSION_ENTRY: &str = "decommission_entry";
|
||||
const POOL_ACTIVATION_FLEET_PROOF_REQUIRED: &str = "pool activation requires a live fleet capability proof";
|
||||
const POOL_ACTIVATION_FLEET_PROOF_EXPIRED: &str = "pool activation fleet capability proof expired before commit";
|
||||
const DECOMMISSION_STAGE_MIGRATE_OBJECT: &str = "migrate_object";
|
||||
const DECOMMISSION_STAGE_CLEANUP_PREFLIGHT: &str = "cleanup_preflight";
|
||||
const DECOMMISSION_STAGE_SOURCE_CLEANUP: &str = "source_cleanup";
|
||||
@@ -1832,6 +1834,13 @@ pub(crate) struct PoolRebalanceActivationFence {
|
||||
}
|
||||
|
||||
impl PoolRebalanceActivationFence {
|
||||
pub(crate) fn set_fleet_proof(
|
||||
&mut self,
|
||||
fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
|
||||
) {
|
||||
self.fleet_proof = fleet_proof;
|
||||
}
|
||||
|
||||
pub(crate) fn ensure_held(&self) -> Result<()> {
|
||||
#[cfg(test)]
|
||||
let forced_lost = self.forced_lost.load(Ordering::Acquire);
|
||||
@@ -1845,7 +1854,7 @@ impl PoolRebalanceActivationFence {
|
||||
.as_ref()
|
||||
.is_some_and(|proof| !crate::services::notification_sys::cross_pool_fence_fleet_proof_matches(proof))
|
||||
{
|
||||
return Err(Error::other("pool activation fleet capability proof expired before commit"));
|
||||
return Err(Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED));
|
||||
}
|
||||
|
||||
Ok(())
|
||||
@@ -1901,7 +1910,17 @@ pub(crate) async fn acquire_pool_activation_fleet_proof(
|
||||
}
|
||||
crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof()
|
||||
.map(Some)
|
||||
.ok_or_else(|| Error::other("pool activation requires a live fleet capability proof"))
|
||||
.ok_or_else(|| Error::other(POOL_ACTIVATION_FLEET_PROOF_REQUIRED))
|
||||
}
|
||||
|
||||
pub(crate) fn is_pool_activation_fleet_proof_error(err: &Error) -> bool {
|
||||
// Save-stage helpers add context by formatting the original error, so the
|
||||
// marker may be nested in the display string. Restrict matching to the
|
||||
// `Error::other` I/O shape used by this activation path.
|
||||
matches!(err, Error::Io(io_error) if io_error.kind() == std::io::ErrorKind::Other && {
|
||||
let message = io_error.to_string();
|
||||
message.contains(POOL_ACTIVATION_FLEET_PROOF_REQUIRED) || message.contains(POOL_ACTIVATION_FLEET_PROOF_EXPIRED)
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -10828,6 +10847,15 @@ mod tests {
|
||||
use crate::bucket::replication::{ReplicationState, ReplicationStatusType};
|
||||
use serde::Serialize;
|
||||
|
||||
#[test]
|
||||
fn pool_activation_fleet_proof_error_classifier_matches_only_retryable_proof_failures() {
|
||||
assert!(is_pool_activation_fleet_proof_error(&Error::other(POOL_ACTIVATION_FLEET_PROOF_REQUIRED)));
|
||||
assert!(is_pool_activation_fleet_proof_error(&Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED)));
|
||||
let wrapped = format!("rebalance meta save failed during start_rebalance: {POOL_ACTIVATION_FLEET_PROOF_EXPIRED}");
|
||||
assert!(is_pool_activation_fleet_proof_error(&Error::other(wrapped)));
|
||||
assert!(!is_pool_activation_fleet_proof_error(&Error::ConfigNotFound));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn decommission_activation_fence_loss_after_durable_save_blocks_publication() {
|
||||
|
||||
@@ -15,7 +15,8 @@
|
||||
use crate::error::{Error, Result};
|
||||
use crate::runtime::sources::{self as runtime_sources, WorkloadSnapshotProviderRef};
|
||||
use metrics::{counter, histogram};
|
||||
use rustfs_concurrency::{AdmissionState, WorkloadAdmissionSnapshotProvider, WorkloadClass};
|
||||
use rustfs_concurrency::WorkloadAdmissionSnapshotProvider;
|
||||
use rustfs_concurrency::workload::ForegroundPressure;
|
||||
use std::time::{Duration, Instant};
|
||||
use tokio::time::sleep;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
@@ -137,23 +138,6 @@ async fn wait_for_data_movement_admission_with_provider(
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
struct ForegroundPressure {
|
||||
class: WorkloadClass,
|
||||
usage_pct: usize,
|
||||
threshold_pct: usize,
|
||||
}
|
||||
|
||||
impl ForegroundPressure {
|
||||
const fn reason(self) -> &'static str {
|
||||
match self.class {
|
||||
WorkloadClass::ForegroundRead => "foreground_read_pressure",
|
||||
WorkloadClass::ForegroundWrite => "foreground_write_pressure",
|
||||
_ => "foreground_pressure",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn foreground_pressure(
|
||||
config: &DataMovementBackpressureConfig,
|
||||
provider: Option<&(dyn WorkloadAdmissionSnapshotProvider + Send + Sync)>,
|
||||
@@ -163,39 +147,11 @@ fn foreground_pressure(
|
||||
}
|
||||
|
||||
let snapshot = provider?.workload_admission_snapshot();
|
||||
[
|
||||
(WorkloadClass::ForegroundRead, config.foreground_read_high_percent),
|
||||
(WorkloadClass::ForegroundWrite, config.foreground_write_high_percent),
|
||||
]
|
||||
.into_iter()
|
||||
.filter_map(|(class, threshold_pct)| {
|
||||
if threshold_pct == 0 {
|
||||
return None;
|
||||
}
|
||||
|
||||
let entry = snapshot.get(class)?;
|
||||
let usage_pct = if matches!(entry.state, AdmissionState::Saturated) {
|
||||
100
|
||||
} else {
|
||||
let limit = entry.limit?;
|
||||
if limit == 0 {
|
||||
return None;
|
||||
}
|
||||
entry
|
||||
.active
|
||||
.unwrap_or(0)
|
||||
.saturating_mul(100)
|
||||
.checked_div(limit)
|
||||
.unwrap_or(100)
|
||||
};
|
||||
|
||||
(usage_pct >= threshold_pct).then_some(ForegroundPressure {
|
||||
class,
|
||||
usage_pct,
|
||||
threshold_pct,
|
||||
})
|
||||
})
|
||||
.max_by_key(|pressure| pressure.usage_pct)
|
||||
rustfs_concurrency::workload::foreground_pressure(
|
||||
&snapshot,
|
||||
config.foreground_read_high_percent,
|
||||
config.foreground_write_high_percent,
|
||||
)
|
||||
}
|
||||
|
||||
fn record_delay_start(
|
||||
@@ -276,7 +232,7 @@ fn record_delay_completion(
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use rustfs_concurrency::{WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot};
|
||||
use rustfs_concurrency::{AdmissionState, WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot, WorkloadClass};
|
||||
use std::sync::Arc;
|
||||
|
||||
#[derive(Debug)]
|
||||
|
||||
@@ -1161,11 +1161,45 @@ fn select_admin_data_usage_snapshot(
|
||||
authoritative.usage_snapshot_converged = Some(true);
|
||||
}
|
||||
match observed {
|
||||
Some(observed)
|
||||
if observed.usage_snapshot_partial
|
||||
&& authoritative.is_complete_bucket_usage_snapshot()
|
||||
&& observed_data_usage_is_newer(&observed, &authoritative) =>
|
||||
{
|
||||
(merge_partial_observation_for_admin(authoritative, observed), true)
|
||||
}
|
||||
Some(observed) if observed_data_usage_is_newer(&observed, &authoritative) => (observed, true),
|
||||
_ => (authoritative, authoritative_format),
|
||||
}
|
||||
}
|
||||
|
||||
fn merge_partial_observation_for_admin(mut authoritative: DataUsageInfo, observed: DataUsageInfo) -> DataUsageInfo {
|
||||
for (bucket, usage) in observed.buckets_usage {
|
||||
authoritative.buckets_usage.insert(bucket, usage);
|
||||
}
|
||||
|
||||
authoritative.last_update = observed.last_update;
|
||||
authoritative.scanner_cycle = observed.scanner_cycle;
|
||||
authoritative.scanner_epoch = observed.scanner_epoch;
|
||||
authoritative.usage_snapshot_complete = false;
|
||||
authoritative.usage_snapshot_partial = true;
|
||||
authoritative.usage_snapshot_converged = Some(false);
|
||||
authoritative.usage_snapshot_authoritative_baseline = observed.usage_snapshot_authoritative_baseline;
|
||||
authoritative.usage_snapshot_set_states = observed.usage_snapshot_set_states;
|
||||
authoritative.usage_snapshot_bootstrap_pending = false;
|
||||
authoritative.buckets_count = authoritative.buckets_usage.len() as u64;
|
||||
authoritative.bucket_sizes = authoritative
|
||||
.buckets_usage
|
||||
.iter()
|
||||
.map(|(bucket, usage)| (bucket.clone(), usage.size))
|
||||
.collect();
|
||||
authoritative.replication_info.clear();
|
||||
authoritative.tier_stats = None;
|
||||
authoritative.unknown_tier_stats = None;
|
||||
authoritative.calculate_totals();
|
||||
authoritative
|
||||
}
|
||||
|
||||
async fn load_admin_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
|
||||
let (authoritative, source) = load_data_usage_snapshot(store.clone()).await?;
|
||||
let observed = load_observed_data_usage_snapshot(store).await;
|
||||
@@ -3217,6 +3251,108 @@ mod tests {
|
||||
assert_eq!(selected.buckets_usage.get("bucket").map(|usage| usage.size), Some(100));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn partial_admin_observation_preserves_authoritative_cold_buckets() {
|
||||
let baseline_time = SystemTime::UNIX_EPOCH + Duration::from_secs(10);
|
||||
let mut authoritative = data_usage_info_for_test("cold", 152_318, 80 * 1024 * 1024 * 1024, baseline_time);
|
||||
authoritative.scanner_epoch = Some(4);
|
||||
authoritative.scanner_cycle = Some(10);
|
||||
authoritative.buckets_usage.insert(
|
||||
"hot".to_string(),
|
||||
BucketUsageInfo {
|
||||
objects_count: 3_000,
|
||||
versions_count: 3_000,
|
||||
size: 400 * 1024 * 1024,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
authoritative.buckets_count = 2;
|
||||
authoritative.bucket_sizes = authoritative
|
||||
.buckets_usage
|
||||
.iter()
|
||||
.map(|(bucket, usage)| (bucket.clone(), usage.size))
|
||||
.collect();
|
||||
authoritative.calculate_totals();
|
||||
authoritative.replication_info.insert(
|
||||
"stale-target".to_string(),
|
||||
BucketTargetUsageInfo {
|
||||
replicated_size: 400 * 1024 * 1024,
|
||||
replicated_count: 3_000,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
authoritative.tier_stats = Some(rustfs_data_usage::AllTierStats {
|
||||
tiers: HashMap::from([(
|
||||
"WARM".to_string(),
|
||||
rustfs_data_usage::TierStats {
|
||||
total_size: 80 * 1024 * 1024 * 1024,
|
||||
num_versions: 152_318,
|
||||
num_objects: 152_318,
|
||||
},
|
||||
)]),
|
||||
});
|
||||
|
||||
let mut observed = DataUsageInfo {
|
||||
last_update: Some(baseline_time + Duration::from_secs(1)),
|
||||
scanner_epoch: Some(4),
|
||||
scanner_cycle: Some(11),
|
||||
usage_snapshot_complete: false,
|
||||
usage_snapshot_partial: true,
|
||||
usage_snapshot_converged: Some(false),
|
||||
usage_snapshot_authoritative_baseline: Some(authoritative.snapshot_identity()),
|
||||
usage_snapshot_set_states: vec![rustfs_data_usage::DataUsageSnapshotSetState {
|
||||
pool_index: 0,
|
||||
set_index: 0,
|
||||
scanner_cycle: Some(11),
|
||||
scanner_epoch: Some(4),
|
||||
scan_plan_digest: Some([1; 32]),
|
||||
complete: true,
|
||||
tombstone: false,
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
observed.buckets_usage.insert(
|
||||
"hot".to_string(),
|
||||
BucketUsageInfo {
|
||||
objects_count: 34,
|
||||
versions_count: 34,
|
||||
size: 8 * 1024 * 1024,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
observed.buckets_count = 1;
|
||||
observed.bucket_sizes.insert("hot".to_string(), 8 * 1024 * 1024);
|
||||
observed.calculate_totals();
|
||||
|
||||
let (selected, current_format) = select_admin_data_usage_snapshot(authoritative, true, Some(observed));
|
||||
|
||||
assert!(current_format);
|
||||
assert!(!selected.usage_snapshot_complete);
|
||||
assert!(selected.usage_snapshot_partial);
|
||||
assert!(selected.is_valid_partial_snapshot());
|
||||
assert_eq!(selected.usage_snapshot_converged, Some(false));
|
||||
assert_eq!(selected.buckets_count, 2);
|
||||
assert_eq!(
|
||||
selected
|
||||
.buckets_usage
|
||||
.get("cold")
|
||||
.map(|usage| (usage.objects_count, usage.size)),
|
||||
Some((152_318, 80 * 1024 * 1024 * 1024))
|
||||
);
|
||||
assert_eq!(
|
||||
selected
|
||||
.buckets_usage
|
||||
.get("hot")
|
||||
.map(|usage| (usage.objects_count, usage.size)),
|
||||
Some((34, 8 * 1024 * 1024))
|
||||
);
|
||||
assert_eq!(selected.objects_total_count, 152_352);
|
||||
assert_eq!(selected.objects_total_size, 80 * 1024 * 1024 * 1024 + 8 * 1024 * 1024);
|
||||
assert!(selected.replication_info.is_empty());
|
||||
assert!(selected.tier_stats.is_none());
|
||||
assert!(selected.unknown_tier_stats.is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn authoritative_save_cleanup_removes_observed_snapshot_best_effort() {
|
||||
let store = UsageCasStore::default();
|
||||
|
||||
@@ -22,6 +22,7 @@ pub type Error = DiskError;
|
||||
pub type Result<T> = core::result::Result<T, Error>;
|
||||
|
||||
const METACACHE_OUTPUT_STREAM_CLOSED: &str = "metacache output stream closed";
|
||||
pub(crate) const HEAL_DANGLING_DELETE_GRACE_MESSAGE: &str = "dangling object deletion deferred by heal grace window";
|
||||
|
||||
/// Marker carried by a shard-read `io::Error` when the underlying reader can
|
||||
/// no longer be realigned after a fresh remote open failed. The marker is
|
||||
@@ -33,6 +34,12 @@ pub(crate) struct TerminalReadError {
|
||||
source: DiskError,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct DanglingDeleteGraceError {
|
||||
retry_after_secs: i64,
|
||||
grace_secs: i64,
|
||||
}
|
||||
|
||||
// DiskError == StorageErr
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum DiskError {
|
||||
@@ -200,6 +207,18 @@ impl StdError for TerminalReadError {
|
||||
}
|
||||
}
|
||||
|
||||
impl std::fmt::Display for DanglingDeleteGraceError {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
write!(
|
||||
f,
|
||||
"{HEAL_DANGLING_DELETE_GRACE_MESSAGE}; retry_after_secs={}; grace_secs={}",
|
||||
self.retry_after_secs, self.grace_secs
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
impl StdError for DanglingDeleteGraceError {}
|
||||
|
||||
fn classify_internode_missing_error(error: &InternodeHttpError) -> Option<DiskError> {
|
||||
if error.is_remote_file_not_found() {
|
||||
return Some(DiskError::FileNotFound);
|
||||
@@ -253,6 +272,24 @@ impl DiskError {
|
||||
DiskError::Io(std::io::Error::other(error))
|
||||
}
|
||||
|
||||
pub(crate) fn dangling_delete_grace(retry_after_secs: i64, grace_secs: i64) -> Self {
|
||||
DiskError::other(DanglingDeleteGraceError {
|
||||
retry_after_secs,
|
||||
grace_secs,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn is_dangling_delete_grace(&self) -> bool {
|
||||
matches!(self, DiskError::Io(io_error) if Self::io_error_is_dangling_delete_grace(io_error))
|
||||
}
|
||||
|
||||
pub fn io_error_is_dangling_delete_grace(io_error: &io::Error) -> bool {
|
||||
io_error
|
||||
.get_ref()
|
||||
.is_some_and(|source| source.downcast_ref::<DanglingDeleteGraceError>().is_some())
|
||||
|| io_error.to_string().contains(HEAL_DANGLING_DELETE_GRACE_MESSAGE)
|
||||
}
|
||||
|
||||
pub(crate) fn metacache_output_stream_closed() -> Self {
|
||||
DiskError::Io(std::io::Error::new(std::io::ErrorKind::BrokenPipe, METACACHE_OUTPUT_STREAM_CLOSED))
|
||||
}
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use crate::config::storageclass::DEFAULT_INLINE_BLOCK;
|
||||
use crate::crash_inject::{self, CrashPoint};
|
||||
use crate::data_usage::local_snapshot::ensure_data_usage_layout;
|
||||
use crate::diagnostics::get::{
|
||||
@@ -10410,8 +10409,14 @@ impl DiskAPI for LocalDisk {
|
||||
fi.data = None;
|
||||
}
|
||||
|
||||
let inline = fi.transition_status.is_empty() && fi.data_dir.is_some() && fi.parts.len() == 1;
|
||||
if inline && fi.shard_file_size(fi.parts[0].actual_size) < DEFAULT_INLINE_BLOCK as i64 {
|
||||
// Keep this compatibility read-ahead decision on the same policy
|
||||
// as PUT's inline admission. In particular, do not use the old
|
||||
// fixed 128 KiB shard limit: a non-inline object in a wider EC
|
||||
// layout can have a smaller shard and would otherwise be copied
|
||||
// out of part.1 during every metadata read. Such objects remain
|
||||
// fully readable through the normal EC reader below.
|
||||
let storage_class_config = runtime_sources::storage_class_config_snapshot();
|
||||
if should_read_legacy_inline_part(&fi, storage_class_config.as_ref()) {
|
||||
let part_path = path_join_buf(&[
|
||||
path,
|
||||
fi.data_dir.map_or_else(|| "".to_string(), |dir| dir.to_string()).as_str(),
|
||||
@@ -10913,6 +10918,21 @@ impl DiskAPI for LocalDisk {
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether a legacy object without the inline marker should have its external
|
||||
/// part materialized into `FileInfo.data` for compatibility with the old GET
|
||||
/// fast path. The marker-bearing path is handled by `read_raw`/`get_file_info`;
|
||||
/// this is only a conservative fallback for old metadata.
|
||||
fn should_read_legacy_inline_part(fi: &FileInfo, storage_class_config: &crate::config::storageclass::Config) -> bool {
|
||||
if !fi.transition_status.is_empty() || fi.data_dir.is_none() || fi.parts.len() != 1 || fi.inline_data() {
|
||||
return false;
|
||||
}
|
||||
|
||||
let part = &fi.parts[0];
|
||||
let shard_size = fi.shard_file_size(part.actual_size);
|
||||
let versioned = fi.versioned || fi.version_id.is_some_and(|version_id| !version_id.is_nil());
|
||||
storage_class_config.should_inline(shard_size, fi.erasure.data_blocks, versioned)
|
||||
}
|
||||
|
||||
impl LocalDisk {
|
||||
pub(crate) async fn rename_data_borrowed(
|
||||
&self,
|
||||
@@ -11049,6 +11069,46 @@ mod test {
|
||||
file_info
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn legacy_inline_read_ahead_matches_writer_policy_for_ec_layouts() {
|
||||
let config = crate::config::storageclass::Config::default();
|
||||
let object_sizes = [128 * 1024_i64, 256 * 1024, 512 * 1024, 1024 * 1024, 4 * 1024 * 1024];
|
||||
|
||||
for (data_shards, parity_shards) in [(8, 4), (12, 4)] {
|
||||
for object_size in object_sizes {
|
||||
let mut fi = FileInfo::new("object", data_shards, parity_shards);
|
||||
fi.data_dir = Some(Uuid::from_u128(1));
|
||||
fi.parts = vec![ObjectPartInfo {
|
||||
number: 1,
|
||||
size: usize::try_from(object_size).expect("test object size should fit usize"),
|
||||
actual_size: object_size,
|
||||
..Default::default()
|
||||
}];
|
||||
|
||||
let writer_decision = config.should_inline(fi.shard_file_size(object_size), data_shards, false);
|
||||
assert_eq!(
|
||||
should_read_legacy_inline_part(&fi, &config),
|
||||
writer_decision,
|
||||
"legacy read-ahead must match PUT for EC{data_shards}+{parity_shards}, size={object_size}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
let mut ec12 = FileInfo::new("object", 12, 4);
|
||||
ec12.data_dir = Some(Uuid::from_u128(1));
|
||||
ec12.parts = vec![ObjectPartInfo {
|
||||
number: 1,
|
||||
size: 1024 * 1024,
|
||||
actual_size: 1024 * 1024,
|
||||
..Default::default()
|
||||
}];
|
||||
assert!(
|
||||
ec12.shard_file_size(1024 * 1024) < crate::config::storageclass::DEFAULT_INLINE_BLOCK as i64,
|
||||
"the regression guard must exercise the old fixed 128 KiB read-ahead boundary"
|
||||
);
|
||||
assert!(!should_read_legacy_inline_part(&ec12, &config));
|
||||
}
|
||||
|
||||
fn test_meta(fi: FileInfo) -> Vec<u8> {
|
||||
let mut meta = FileMeta::default();
|
||||
meta.add_version(fi).expect("test metadata should accept file info");
|
||||
|
||||
@@ -27,6 +27,8 @@ use std::io;
|
||||
use std::io::ErrorKind;
|
||||
use std::pin::Pin;
|
||||
use std::sync::Mutex;
|
||||
#[cfg(test)]
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::task::{Context, Poll, ready};
|
||||
use std::time::Instant;
|
||||
use tokio::io::{AsyncRead, ReadBuf};
|
||||
@@ -38,6 +40,14 @@ const DEFAULT_RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT: usize = 2;
|
||||
const FILL_POLICY_SINGLE_INFLIGHT: &str = "single_inflight";
|
||||
const FILL_POLICY_DUAL_INFLIGHT: &str = "dual_inflight";
|
||||
|
||||
#[cfg(test)]
|
||||
static SINGLE_INFLIGHT_CONSTRUCTIONS: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn test_single_inflight_construction_count() -> u64 {
|
||||
SINGLE_INFLIGHT_CONSTRUCTIONS.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
type FillTask = oneshot::Receiver<FillResult>;
|
||||
|
||||
struct FillWorker {
|
||||
@@ -155,6 +165,23 @@ where
|
||||
Self::new_with_fill_policy_inner(source, engine, total_length, metrics_path, FillPolicy::from_env())
|
||||
}
|
||||
|
||||
/// Construct the bounded reader without lookahead.
|
||||
///
|
||||
/// Mid-size GETs are latency-sensitive and are already gated to a single
|
||||
/// plain part. Keeping one stripe in flight avoids retaining a second
|
||||
/// decoded output buffer while preserving the same source, reconstruction,
|
||||
/// bitrot and cancellation semantics as the general streaming reader.
|
||||
pub(crate) fn new_single_inflight_with_metrics_path(
|
||||
source: S,
|
||||
engine: E,
|
||||
total_length: usize,
|
||||
metrics_path: &'static str,
|
||||
) -> io::Result<Self> {
|
||||
#[cfg(test)]
|
||||
SINGLE_INFLIGHT_CONSTRUCTIONS.fetch_add(1, Ordering::Relaxed);
|
||||
Self::new_with_fill_policy_inner(source, engine, total_length, metrics_path, FillPolicy::SingleInFlight)
|
||||
}
|
||||
|
||||
fn new_with_fill_policy_inner(
|
||||
source: S,
|
||||
engine: E,
|
||||
@@ -602,7 +629,8 @@ where
|
||||
|
||||
loop {
|
||||
if self.output_pos < self.output_buf.len() {
|
||||
if self.prefetched_bufs.len() < self.fill_policy.max_inflight()
|
||||
if self.fill_policy == FillPolicy::DualInFlight
|
||||
&& self.prefetched_bufs.len() < self.fill_policy.max_inflight()
|
||||
&& self.prefetch_error.is_none()
|
||||
&& self.remaining > 0
|
||||
&& let Poll::Ready(result) = self.poll_prefetch(cx)
|
||||
@@ -1620,6 +1648,149 @@ mod tests {
|
||||
assert_eq!(decoded, data);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn single_inflight_reader_reads_full_body_without_lookahead() {
|
||||
let erasure = Erasure::new(4, 2, 32);
|
||||
let data = (0..96u8).collect::<Vec<_>>();
|
||||
let read_count = Arc::new(AtomicUsize::new(0));
|
||||
let mut source = source_from_data(&erasure, &data, &[]);
|
||||
source.read_count = Some(Arc::clone(&read_count));
|
||||
let engine = LegacyEcDecodeEngine::new(erasure);
|
||||
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
|
||||
source,
|
||||
engine,
|
||||
data.len(),
|
||||
GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
)
|
||||
.expect("single-inflight reader should be constructed");
|
||||
let mut decoded = Vec::new();
|
||||
|
||||
reader
|
||||
.read_to_end(&mut decoded)
|
||||
.await
|
||||
.expect("single-inflight reader should decode the complete body");
|
||||
|
||||
assert_eq!(decoded, data);
|
||||
assert_eq!(
|
||||
read_count.load(Ordering::SeqCst),
|
||||
3,
|
||||
"single-inflight must not read ahead after the final stripe"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn single_inflight_reader_preserves_partial_reads() {
|
||||
let erasure = Erasure::new(4, 2, 32);
|
||||
let data = (0..83u8).collect::<Vec<_>>();
|
||||
let engine = LegacyEcDecodeEngine::new(erasure.clone());
|
||||
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
|
||||
source_from_data(&erasure, &data, &[]),
|
||||
engine,
|
||||
data.len(),
|
||||
GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
)
|
||||
.expect("single-inflight reader should be constructed");
|
||||
let mut decoded = Vec::with_capacity(data.len());
|
||||
let mut chunk = [0u8; 3];
|
||||
|
||||
loop {
|
||||
let read = reader.read(&mut chunk).await.expect("partial read should succeed");
|
||||
if read == 0 {
|
||||
break;
|
||||
}
|
||||
decoded.extend_from_slice(&chunk[..read]);
|
||||
}
|
||||
|
||||
assert_eq!(decoded, data);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn single_inflight_reader_does_not_prefetch_before_output_is_drained() {
|
||||
let erasure = Erasure::new(4, 2, 32);
|
||||
let data = (0..96u8).collect::<Vec<_>>();
|
||||
let read_count = Arc::new(AtomicUsize::new(0));
|
||||
let mut source = source_from_data(&erasure, &data, &[]);
|
||||
source.read_count = Some(Arc::clone(&read_count));
|
||||
let engine = LegacyEcDecodeEngine::new(erasure);
|
||||
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
|
||||
source,
|
||||
engine,
|
||||
data.len(),
|
||||
GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
)
|
||||
.expect("single-inflight reader should be constructed");
|
||||
let mut first = [0u8; 3];
|
||||
|
||||
reader
|
||||
.read_exact(&mut first)
|
||||
.await
|
||||
.expect("first partial read should succeed");
|
||||
|
||||
assert_eq!(
|
||||
read_count.load(Ordering::SeqCst),
|
||||
1,
|
||||
"single-inflight must not prefetch while output remains"
|
||||
);
|
||||
assert_eq!(&first, &data[..3]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn single_inflight_reader_reconstructs_degraded_body() {
|
||||
let erasure = Erasure::new(4, 2, 32);
|
||||
let data = (0..97u16).map(|value| value as u8).collect::<Vec<_>>();
|
||||
let engine = LegacyEcDecodeEngine::new(erasure.clone());
|
||||
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
|
||||
source_from_data(&erasure, &data, &[1]),
|
||||
engine,
|
||||
data.len(),
|
||||
GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
)
|
||||
.expect("single-inflight reader should be constructed");
|
||||
let mut decoded = Vec::new();
|
||||
|
||||
reader
|
||||
.read_to_end(&mut decoded)
|
||||
.await
|
||||
.expect("a readable degraded stripe should be reconstructed");
|
||||
|
||||
assert_eq!(decoded, data);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn single_inflight_reader_surfaces_error_after_buffered_body() {
|
||||
let erasure = Erasure::new(4, 2, 32);
|
||||
let first_stripe = (0..32u8).collect::<Vec<_>>();
|
||||
let first_state = source_from_data(&erasure, &first_stripe, &[])
|
||||
.stripes
|
||||
.pop_front()
|
||||
.expect("first stripe should exist");
|
||||
let source = VecStripeSource {
|
||||
stripes: VecDeque::from([
|
||||
first_state,
|
||||
StripeReadState::from_parts(Vec::new(), Vec::new(), erasure.data_shards),
|
||||
]),
|
||||
read_quorum: erasure.data_shards,
|
||||
read_count: None,
|
||||
};
|
||||
let engine = LegacyEcDecodeEngine::new(erasure);
|
||||
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
|
||||
source,
|
||||
engine,
|
||||
first_stripe.len() + 1,
|
||||
GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
)
|
||||
.expect("single-inflight reader should be constructed");
|
||||
let mut decoded = Vec::new();
|
||||
|
||||
let error = reader
|
||||
.read_to_end(&mut decoded)
|
||||
.await
|
||||
.expect_err("short source error should be returned after buffered bytes");
|
||||
|
||||
assert_eq!(error.kind(), ErrorKind::Other);
|
||||
assert_eq!(decoded, first_stripe);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn erasure_decode_reader_stops_at_eof_for_empty_object() {
|
||||
let erasure = Erasure::new(4, 2, 32);
|
||||
@@ -1882,14 +2053,9 @@ mod tests {
|
||||
};
|
||||
let engine = LegacyEcDecodeEngine::new(Erasure::new(1, 0, 32));
|
||||
let task = tokio::spawn(async move {
|
||||
let mut reader = ErasureDecodeReader::new_with_fill_policy(
|
||||
source,
|
||||
engine,
|
||||
1,
|
||||
GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
FillPolicy::SingleInFlight,
|
||||
)
|
||||
.expect("reader should be constructed");
|
||||
let mut reader =
|
||||
ErasureDecodeReader::new_single_inflight_with_metrics_path(source, engine, 1, GET_OBJECT_PATH_CODEC_STREAMING)
|
||||
.expect("reader should be constructed");
|
||||
let mut first_read = [0u8; 1];
|
||||
let _ = reader.read(&mut first_read).await;
|
||||
});
|
||||
@@ -2226,7 +2392,7 @@ mod tests {
|
||||
engine,
|
||||
data.len(),
|
||||
GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
FillPolicy::SingleInFlight,
|
||||
FillPolicy::DualInFlight,
|
||||
)
|
||||
.expect("reader should be constructed");
|
||||
let mut first_read = [0u8; 1];
|
||||
@@ -2235,13 +2401,11 @@ mod tests {
|
||||
|
||||
assert_eq!(read, first_read.len());
|
||||
assert_eq!(first_read[0], data[0]);
|
||||
timeout(Duration::from_secs(1), async {
|
||||
while read_count.load(Ordering::SeqCst) < 2 {
|
||||
yield_now().await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("reader should start reading the next stripe before the current output buffer is fully consumed");
|
||||
assert_eq!(
|
||||
read_count.load(Ordering::SeqCst),
|
||||
2,
|
||||
"dual-inflight reader should prefetch the next stripe before returning the first byte"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -157,6 +157,8 @@ pub enum StorageError {
|
||||
InvalidPartNumber(usize),
|
||||
#[error("Your proposed upload is smaller than the minimum allowed size. Part {0} size {1} is less than minimum {2}")]
|
||||
EntityTooSmall(usize, i64, i64),
|
||||
#[error("multipart upload size {0} exceeds the configured limit {1}")]
|
||||
EntityTooLarge(u64, u64),
|
||||
|
||||
// ── Erasure / Quorum ─────────────────────────────────────────────
|
||||
#[error("erasure read quorum")]
|
||||
@@ -275,6 +277,10 @@ impl StorageError {
|
||||
| StorageError::NamespaceLockQuorumUnavailable { .. }
|
||||
)
|
||||
}
|
||||
|
||||
pub fn is_dangling_delete_grace(&self) -> bool {
|
||||
matches!(self, StorageError::Io(io_error) if DiskError::io_error_is_dangling_delete_grace(io_error))
|
||||
}
|
||||
}
|
||||
|
||||
impl From<HTTPRangeError> for StorageError {
|
||||
@@ -554,6 +560,7 @@ impl Clone for StorageError {
|
||||
StorageError::DecommissionNotStarted => StorageError::DecommissionNotStarted,
|
||||
StorageError::InvalidPart(a, b, c) => StorageError::InvalidPart(*a, b.clone(), c.clone()),
|
||||
StorageError::EntityTooSmall(a, b, c) => StorageError::EntityTooSmall(*a, *b, *c),
|
||||
StorageError::EntityTooLarge(a, b) => StorageError::EntityTooLarge(*a, *b),
|
||||
StorageError::DoneForNow => StorageError::DoneForNow,
|
||||
StorageError::DecommissionAlreadyRunning => StorageError::DecommissionAlreadyRunning,
|
||||
StorageError::RebalanceAlreadyRunning => StorageError::RebalanceAlreadyRunning,
|
||||
@@ -673,6 +680,7 @@ impl StorageError {
|
||||
StorageError::InsufficientWriteQuorum(_, _) => StorageErrorCode::InsufficientWriteQuorum,
|
||||
StorageError::PreconditionFailed => StorageErrorCode::PreconditionFailed,
|
||||
StorageError::EntityTooSmall(_, _, _) => StorageErrorCode::EntityTooSmall,
|
||||
StorageError::EntityTooLarge(_, _) => StorageErrorCode::EntityTooLarge,
|
||||
StorageError::InvalidRangeSpec(_) => StorageErrorCode::InvalidRangeSpec,
|
||||
StorageError::NotModified => StorageErrorCode::NotModified,
|
||||
StorageError::InvalidPartNumber(_) => StorageErrorCode::InvalidPartNumber,
|
||||
@@ -795,6 +803,7 @@ impl StorageError {
|
||||
StorageErrorCode::EntityTooSmall => {
|
||||
Some(StorageError::EntityTooSmall(Default::default(), Default::default(), Default::default()))
|
||||
}
|
||||
StorageErrorCode::EntityTooLarge => Some(StorageError::EntityTooLarge(Default::default(), Default::default())),
|
||||
StorageErrorCode::InvalidRangeSpec => Some(StorageError::InvalidRangeSpec(Default::default())),
|
||||
StorageErrorCode::NotModified => Some(StorageError::NotModified),
|
||||
StorageErrorCode::InvalidPartNumber => Some(StorageError::InvalidPartNumber(Default::default())),
|
||||
|
||||
@@ -730,7 +730,18 @@ impl ReadPlan {
|
||||
})
|
||||
.await
|
||||
.map_err(Error::other)?
|
||||
.ok_or_else(|| Error::other("encrypted object metadata is incomplete"))?;
|
||||
.ok_or_else(|| {
|
||||
// The resolver saw no encryption it recognizes, yet the
|
||||
// object's markers say it is encrypted. Keep failing closed,
|
||||
// but as a typed, non-retryable error: the condition is a
|
||||
// permanent property of the stored metadata, not a fault a
|
||||
// retry can fix.
|
||||
Error::other(EncryptionResolutionError::new(
|
||||
EncryptionResolutionErrorKind::InvalidMetadata,
|
||||
"object is marked encrypted, but no decryption material could be resolved from its metadata; \
|
||||
the encryption metadata is incomplete or in a format this server cannot read",
|
||||
))
|
||||
})?;
|
||||
let material = resolved;
|
||||
#[cfg(feature = "rio-v2")]
|
||||
let uses_legacy_encryption = matches!(material.mode, ReadEncryptionMode::Direct { .. });
|
||||
|
||||
@@ -752,12 +752,18 @@ impl ObjectInfo {
|
||||
.any(|key| rustfs_utils::http::is_object_encryption_marker(key))
|
||||
}
|
||||
|
||||
/// Maximum inline size for non-versioned objects (128 KiB).
|
||||
/// Matches `DEFAULT_INLINE_BLOCK` in `storageclass.rs`.
|
||||
/// Historical non-versioned inline size reference.
|
||||
///
|
||||
/// Inline admission is layout-specific now; callers must not use this
|
||||
/// constant to decide whether an object is eligible for the fast path.
|
||||
#[deprecated(note = "inline eligibility is layout-specific; use persisted metadata and the read-path policy")]
|
||||
pub const INLINE_MAX_SIZE: i64 = 128 * 1024;
|
||||
|
||||
/// Maximum inline size for versioned objects (16 KiB).
|
||||
/// Matches `DEFAULT_INLINE_BLOCK / 8` in `storageclass.rs`.
|
||||
/// Historical versioned inline size reference.
|
||||
///
|
||||
/// Inline admission is layout-specific now; callers must not use this
|
||||
/// constant to decide whether an object is eligible for the fast path.
|
||||
#[deprecated(note = "inline eligibility is layout-specific; use persisted metadata and the read-path policy")]
|
||||
pub const INLINE_MAX_SIZE_VERSIONED: i64 = 16 * 1024;
|
||||
|
||||
/// Returns `true` when this object qualifies for the inline data fast path.
|
||||
@@ -765,10 +771,12 @@ impl ObjectInfo {
|
||||
/// The inline fast path decodes erasure-coded data entirely in memory,
|
||||
/// bypassing disk I/O, duplex pipes, and the disk-read semaphore.
|
||||
///
|
||||
/// The `inlined` flag is the primary signal — PUT sets it through the
|
||||
/// captured storage-class snapshot's `Config::should_inline`, which applies
|
||||
/// the correct version-aware threshold (128 KiB non-versioned, 16 KiB versioned).
|
||||
/// The size check below is a safety net using the same thresholds.
|
||||
/// The persisted `inlined` flag is the canonical size-policy decision. PUT
|
||||
/// sets it through the captured storage-class snapshot's effective policy,
|
||||
/// which is layout- and version-aware. Reapplying a fixed object-size limit
|
||||
/// here would disagree with that policy for wider EC layouts and explicit
|
||||
/// inline configurations. The direct-memory reader retains its own bounded
|
||||
/// 128 KiB allocation gate at the call site.
|
||||
///
|
||||
/// Additional conditions:
|
||||
/// - Single part
|
||||
@@ -779,14 +787,8 @@ impl ObjectInfo {
|
||||
if !self.inlined {
|
||||
return false;
|
||||
}
|
||||
// Apply the same version-aware threshold as PUT (storageclass.rs).
|
||||
let max_size = if self.version_id.is_some() {
|
||||
Self::INLINE_MAX_SIZE_VERSIONED
|
||||
} else {
|
||||
Self::INLINE_MAX_SIZE
|
||||
};
|
||||
self.parts.len() == 1
|
||||
&& self.size <= max_size
|
||||
&& self.size >= 0
|
||||
&& !self.is_encrypted()
|
||||
&& !self.is_compressed()
|
||||
&& self.transitioned_object.tier.is_empty()
|
||||
@@ -1382,14 +1384,14 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inline_fast_path_eligibility_preserves_exact_versioned_boundaries() {
|
||||
fn inline_fast_path_eligibility_follows_persisted_marker() {
|
||||
for (case, size, versioned, expected) in [
|
||||
("unversioned below", 128 * 1024 - 1, false, true),
|
||||
("unversioned exact", 128 * 1024, false, true),
|
||||
("unversioned above", 128 * 1024 + 1, false, false),
|
||||
("unversioned above", 128 * 1024 + 1, false, true),
|
||||
("versioned below", 16 * 1024 - 1, true, true),
|
||||
("versioned exact", 16 * 1024, true, true),
|
||||
("versioned above", 16 * 1024 + 1, true, false),
|
||||
("versioned above", 16 * 1024 + 1, true, true),
|
||||
] {
|
||||
assert_eq!(
|
||||
inline_fast_path_object(size, versioned).is_inline_fast_path_eligible(),
|
||||
@@ -1399,9 +1401,29 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inline_fast_path_marker_allows_ec8_and_ec12_layout_specific_256kib_objects() {
|
||||
for data_blocks in [8, 12] {
|
||||
let object = ObjectInfo {
|
||||
size: 256 * 1024,
|
||||
data_blocks,
|
||||
parity_blocks: 4,
|
||||
inlined: true,
|
||||
version_id: Some(Uuid::from_u128(1)),
|
||||
parts: Arc::new(vec![ObjectPartInfo::default()]),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
assert!(
|
||||
object.is_inline_fast_path_eligible(),
|
||||
"the persisted inline marker must be authoritative for EC{data_blocks}+4"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inline_fast_path_eligibility_rejects_incompatible_object_shapes() {
|
||||
let mut object = inline_fast_path_object(ObjectInfo::INLINE_MAX_SIZE, false);
|
||||
let mut object = inline_fast_path_object(128 * 1024, false);
|
||||
|
||||
object.inlined = false;
|
||||
assert!(!object.is_inline_fast_path_eligible(), "non-inline objects must fall back");
|
||||
|
||||
@@ -234,6 +234,39 @@ pub(crate) fn install_cross_pool_fence_fleet_proof_for_test() {
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) struct CrossPoolFenceFleetProofGuard {
|
||||
previous_proof: Option<FleetCapabilityProof>,
|
||||
previous_topology_conflict: bool,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
impl Drop for CrossPoolFenceFleetProofGuard {
|
||||
fn drop(&mut self) {
|
||||
let mut state = cross_pool_fence_fleet_proof_slot()
|
||||
.write()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
state.proof = self.previous_proof.take();
|
||||
state.topology_conflict = self.previous_topology_conflict;
|
||||
}
|
||||
}
|
||||
|
||||
/// Temporarily revoke the test proof so activation paths can exercise their
|
||||
/// fail-closed behavior without changing the process-wide topology binding.
|
||||
#[cfg(test)]
|
||||
pub(crate) fn without_cross_pool_fence_fleet_proof_for_test() -> CrossPoolFenceFleetProofGuard {
|
||||
let mut state = cross_pool_fence_fleet_proof_slot()
|
||||
.write()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
let guard = CrossPoolFenceFleetProofGuard {
|
||||
previous_proof: state.proof.clone(),
|
||||
previous_topology_conflict: state.topology_conflict,
|
||||
};
|
||||
state.proof = None;
|
||||
state.topology_conflict = true;
|
||||
guard
|
||||
}
|
||||
|
||||
#[cfg(any(test, feature = "test-util"))]
|
||||
pub fn rotate_cross_pool_fence_fleet_proof_for_test() -> bool {
|
||||
let mut state = cross_pool_fence_fleet_proof_slot()
|
||||
|
||||
@@ -570,10 +570,13 @@ impl ECStore {
|
||||
where
|
||||
S: EcstoreObjectIO + StorageNamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
|
||||
{
|
||||
let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
|
||||
// Lock order: pool_meta_save_gate -> pool.bin -> rebalance.bin.
|
||||
let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
|
||||
pool_meta_guard.ensure_write_safe("rebalance worker activation")?;
|
||||
let activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), fleet_proof).await?;
|
||||
// Classify the durable rebalance record while holding both namespace
|
||||
// fences. A terminal record is a no-op and must not depend on the
|
||||
// notification subsystem having published a fleet proof yet.
|
||||
let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?;
|
||||
let pool_meta = self
|
||||
.load_runtime_pool_meta_under_activation_fence(&mut pool_meta_guard, &activation_fence, "rebalance worker activation")
|
||||
.await?;
|
||||
@@ -597,10 +600,17 @@ impl ECStore {
|
||||
}
|
||||
|
||||
activation_fence.ensure_held()?;
|
||||
if !is_rebalance_conflicting_with_decommission(&persisted) {
|
||||
if !crate::services::rebalance::rebalance_requires_worker_activation(&persisted) {
|
||||
return Ok(RebalanceWorkerActivationFence::NotStartedTerminal);
|
||||
}
|
||||
|
||||
// Active worker admission still requires the fail-closed fleet proof.
|
||||
// Attach it immediately before the final fence validation so expiry or
|
||||
// topology changes are checked again at every later commit boundary.
|
||||
let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
|
||||
activation_fence.set_fleet_proof(fleet_proof);
|
||||
activation_fence.ensure_held()?;
|
||||
|
||||
Ok(RebalanceWorkerActivationFence::Ready(Box::new(activation_fence)))
|
||||
}
|
||||
|
||||
@@ -1476,6 +1486,64 @@ mod tests {
|
||||
assert_activation_locks_released(&store).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn rebalance_worker_skips_terminal_metadata_without_fleet_proof() {
|
||||
let rebalance_id = "terminal-metadata-without-proof";
|
||||
let completed = RebalanceMeta {
|
||||
id: rebalance_id.to_string(),
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Completed,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(Some(completed)).await;
|
||||
let _proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
|
||||
|
||||
let activation = store
|
||||
.fence_rebalance_worker_activation(store.pools[0].clone(), rebalance_id)
|
||||
.await
|
||||
.expect("terminal metadata should not require a fleet proof");
|
||||
assert!(matches!(activation, RebalanceWorkerActivationFence::NotStartedTerminal));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn rebalance_worker_still_requires_fleet_proof_for_active_metadata() {
|
||||
let rebalance_id = "active-metadata-without-proof";
|
||||
let active = RebalanceMeta {
|
||||
id: rebalance_id.to_string(),
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Started,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(Some(active)).await;
|
||||
let _proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
|
||||
|
||||
let err = match store
|
||||
.fence_rebalance_worker_activation(store.pools[0].clone(), rebalance_id)
|
||||
.await
|
||||
{
|
||||
Ok(_) => panic!("active metadata must not be admitted without a fleet proof"),
|
||||
Err(err) => err,
|
||||
};
|
||||
assert!(
|
||||
err.to_string()
|
||||
.contains("pool activation requires a live fleet capability proof")
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn rebalance_activation_adopts_commit_after_post_save_fence_loss() {
|
||||
|
||||
@@ -214,6 +214,14 @@ pub(super) fn is_rebalance_in_progress(meta: &RebalanceMeta) -> bool {
|
||||
meta.pool_stats.iter().any(is_rebalance_pool_active)
|
||||
}
|
||||
|
||||
/// Persisted rebalance metadata requires worker activation only while it has
|
||||
/// not reached a durable terminal marker and at least one pool is still marked
|
||||
/// active. Merely finding `rebalance.bin` is not evidence that admission is
|
||||
/// required: terminal metadata is retained for status reporting.
|
||||
pub(crate) fn rebalance_requires_worker_activation(meta: &RebalanceMeta) -> bool {
|
||||
meta.stopped_at.is_none() && is_rebalance_in_progress(meta)
|
||||
}
|
||||
|
||||
pub(crate) fn is_rebalance_conflicting_with_decommission(meta: &RebalanceMeta) -> bool {
|
||||
is_rebalance_in_progress(meta)
|
||||
}
|
||||
|
||||
@@ -49,8 +49,8 @@ mod worker;
|
||||
|
||||
#[cfg(feature = "test-util")]
|
||||
pub use entry::test_util::PausedRebalanceEntryTestFixture;
|
||||
pub(crate) use meta::is_rebalance_conflicting_with_decommission;
|
||||
pub use meta::{decode_rebalance_stop_propagation_record, encode_rebalance_stop_propagation_record};
|
||||
pub(crate) use meta::{is_rebalance_conflicting_with_decommission, rebalance_requires_worker_activation};
|
||||
pub use types::{
|
||||
DiskStat, RebalSaveOpt, RebalStatus, RebalanceCleanupWarningEntry, RebalanceCleanupWarnings, RebalanceInfo, RebalanceMeta,
|
||||
RebalanceStats, RebalanceStopPropagationRecord,
|
||||
|
||||
@@ -22,11 +22,11 @@ use super::meta::{
|
||||
is_rebalance_in_progress, is_rebalance_meta_replaceable_for_new_id, is_rebalance_stopped_terminal_event,
|
||||
mark_rebalance_bucket_done, merge_rebalance_bucket_lists, merge_rebalance_meta, next_rebal_bucket_from_stat,
|
||||
percent_free_ratio, rebalance_goal_reached, rebalance_meta_load_no_data_error, rebalance_meta_load_unknown_format_error,
|
||||
rebalance_meta_load_unknown_version_error, record_rebalance_cleanup_warning_in_meta, remove_rebalanced_buckets_from_queue,
|
||||
resolve_next_rebalance_bucket, resolve_rebalance_participants, should_accept_rebalance_stats_update,
|
||||
should_ignore_rebalance_data_usage_cache, should_pool_participate, should_preserve_rebalance_stopped_state,
|
||||
should_skip_start_rebalance, stop_rebalance_meta_snapshot, stop_rebalance_state, take_bucket_from_rebalance_queue,
|
||||
validate_init_rebalance_state, validate_start_rebalance_state,
|
||||
rebalance_meta_load_unknown_version_error, rebalance_requires_worker_activation, record_rebalance_cleanup_warning_in_meta,
|
||||
remove_rebalanced_buckets_from_queue, resolve_next_rebalance_bucket, resolve_rebalance_participants,
|
||||
should_accept_rebalance_stats_update, should_ignore_rebalance_data_usage_cache, should_pool_participate,
|
||||
should_preserve_rebalance_stopped_state, should_skip_start_rebalance, stop_rebalance_meta_snapshot, stop_rebalance_state,
|
||||
take_bucket_from_rebalance_queue, validate_init_rebalance_state, validate_start_rebalance_state,
|
||||
};
|
||||
use super::migration::{
|
||||
MigrationBackend, MigrationVersionResult, migrate_entry_version, migrate_entry_version_with_retry_wait,
|
||||
@@ -3386,6 +3386,52 @@ fn test_is_rebalance_in_progress_only_started_participants() {
|
||||
assert!(is_rebalance_in_progress(&meta));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_rebalance_requires_worker_activation_only_for_active_non_stopped_metadata() {
|
||||
let now = OffsetDateTime::now_utc();
|
||||
let active = RebalanceMeta {
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Started,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
let stopped_active = RebalanceMeta {
|
||||
stopped_at: Some(now),
|
||||
pool_stats: active.pool_stats.clone(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
assert!(rebalance_requires_worker_activation(&active));
|
||||
for status in [
|
||||
RebalStatus::Completed,
|
||||
RebalStatus::Stopped,
|
||||
RebalStatus::Failed,
|
||||
RebalStatus::None,
|
||||
] {
|
||||
let terminal = RebalanceMeta {
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
assert!(
|
||||
!rebalance_requires_worker_activation(&terminal),
|
||||
"terminal status {status:?} must not resume"
|
||||
);
|
||||
}
|
||||
assert!(!rebalance_requires_worker_activation(&stopped_active));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_is_rebalance_conflicting_with_decommission_true_when_in_progress() {
|
||||
let now = OffsetDateTime::now_utc();
|
||||
|
||||
@@ -65,6 +65,7 @@ use crate::storage_api_contracts::{
|
||||
};
|
||||
use crate::{
|
||||
bucket::lifecycle::{
|
||||
get_lifecycle_config,
|
||||
tier_delete_journal::{TIER_DELETE_JOURNAL_PREFIX, decode_tier_delete_journal_entry},
|
||||
transition_transaction::{TRANSITION_TRANSACTION_RECORD_PREFIX, decode_transition_transaction_record},
|
||||
},
|
||||
@@ -81,7 +82,7 @@ use rustfs_filemeta::FileInfo;
|
||||
use rustfs_rio::HashReader;
|
||||
use rustfs_s3_client::{admin_handler_utils::AdminError, provider_versions::ProviderVersionCapabilities};
|
||||
use rustfs_utils::path::{SLASH_SEPARATOR, path_join};
|
||||
use s3s::S3ErrorCode;
|
||||
use s3s::{S3ErrorCode, dto::BucketLifecycleConfiguration};
|
||||
|
||||
use super::{
|
||||
tier_handlers::{ERR_TIER_BUCKET_NOT_FOUND, ERR_TIER_CONNECT_ERR, ERR_TIER_INVALID_CREDENTIALS, ERR_TIER_PERM_ERR},
|
||||
@@ -535,6 +536,10 @@ enum TierCandidateMutation {
|
||||
}
|
||||
|
||||
impl TierCandidateMutation {
|
||||
fn checks_lifecycle_references(&self) -> bool {
|
||||
!matches!(self, Self::Add(_, true) | Self::Remove(_, true) | Self::Clear(true))
|
||||
}
|
||||
|
||||
fn intent_kind(&self) -> TierMutationIntentKind {
|
||||
match self {
|
||||
Self::Add(_, _) => TierMutationIntentKind::Add,
|
||||
@@ -694,6 +699,7 @@ fn tier_backend_identity_admin_error(err: io::Error) -> AdminError {
|
||||
admin_err
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
trait TierReferenceProofStore:
|
||||
EcstoreObjectIO
|
||||
+ BucketOperations<Error = Error>
|
||||
@@ -705,28 +711,27 @@ trait TierReferenceProofStore:
|
||||
WalkOptions = TierReferenceProofWalkOptions,
|
||||
WalkCancellation = tokio_util::sync::CancellationToken,
|
||||
WalkResultSender = tokio::sync::mpsc::Sender<StorageObjectInfoOrErr<ObjectInfo, Error>>,
|
||||
>
|
||||
> + Send
|
||||
+ Sync
|
||||
{
|
||||
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>>;
|
||||
}
|
||||
|
||||
impl<T> TierReferenceProofStore for T where
|
||||
T: EcstoreObjectIO
|
||||
+ BucketOperations<Error = Error>
|
||||
+ ListOperations<
|
||||
Error = Error,
|
||||
ListObjectsV2Info = StorageListObjectsV2Info<ObjectInfo>,
|
||||
ListObjectVersionsInfo = StorageListObjectVersionsInfo<ObjectInfo>,
|
||||
ObjectInfoOrErr = StorageObjectInfoOrErr<ObjectInfo, Error>,
|
||||
WalkOptions = TierReferenceProofWalkOptions,
|
||||
WalkCancellation = tokio_util::sync::CancellationToken,
|
||||
WalkResultSender = tokio::sync::mpsc::Sender<StorageObjectInfoOrErr<ObjectInfo, Error>>,
|
||||
>
|
||||
{
|
||||
#[async_trait::async_trait]
|
||||
impl TierReferenceProofStore for ECStore {
|
||||
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
|
||||
match get_lifecycle_config(bucket).await {
|
||||
Ok((config, _updated_at)) => Ok(Some(config)),
|
||||
Err(Error::ConfigNotFound) => Ok(None),
|
||||
Err(err) => Err(err),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn ensure_no_authoritative_tier_object_references<S>(
|
||||
api: Arc<S>,
|
||||
affected_targets: &[TierMutationIntentTarget],
|
||||
check_lifecycle_references: bool,
|
||||
) -> std::result::Result<(), AdminError>
|
||||
where
|
||||
S: TierReferenceProofStore,
|
||||
@@ -739,12 +744,13 @@ where
|
||||
if targets.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
ensure_no_authoritative_target_references(api, &targets).await
|
||||
ensure_no_authoritative_target_references(api, &targets, check_lifecycle_references).await
|
||||
}
|
||||
|
||||
async fn ensure_no_authoritative_target_references<S>(
|
||||
api: Arc<S>,
|
||||
targets: &[TierMutationIntentTarget],
|
||||
check_lifecycle_references: bool,
|
||||
) -> std::result::Result<(), AdminError>
|
||||
where
|
||||
S: TierReferenceProofStore,
|
||||
@@ -754,6 +760,9 @@ where
|
||||
.await
|
||||
.map_err(tier_reference_proof_admin_error)?;
|
||||
for bucket in buckets {
|
||||
if check_lifecycle_references {
|
||||
ensure_no_authoritative_lifecycle_references(api.as_ref(), &bucket.name, targets).await?;
|
||||
}
|
||||
let mut marker = None;
|
||||
let mut version_marker = None;
|
||||
loop {
|
||||
@@ -789,6 +798,66 @@ where
|
||||
ensure_no_authoritative_persisted_references(api, targets).await
|
||||
}
|
||||
|
||||
async fn ensure_no_authoritative_lifecycle_references(
|
||||
api: &impl TierReferenceProofStore,
|
||||
bucket: &str,
|
||||
targets: &[TierMutationIntentTarget],
|
||||
) -> std::result::Result<(), AdminError> {
|
||||
match api.lifecycle_config_for_reference_proof(bucket).await {
|
||||
Ok(Some(config)) => {
|
||||
if let Some(reference) = lifecycle_config_target_reference(&config, targets) {
|
||||
return Err(tier_reference_proof_lifecycle_in_use_error(
|
||||
reference.tier_name,
|
||||
bucket,
|
||||
reference.rule_id,
|
||||
));
|
||||
}
|
||||
}
|
||||
Ok(None) => {}
|
||||
Err(err) => {
|
||||
return Err(tier_reference_proof_admin_error(format!("bucket {bucket} lifecycle config: {err}")));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
struct TierLifecycleReference<'a> {
|
||||
tier_name: &'a str,
|
||||
rule_id: Option<&'a str>,
|
||||
}
|
||||
|
||||
fn lifecycle_config_target_reference<'a>(
|
||||
config: &'a BucketLifecycleConfiguration,
|
||||
targets: &[TierMutationIntentTarget],
|
||||
) -> Option<TierLifecycleReference<'a>> {
|
||||
for rule in &config.rules {
|
||||
let rule_id = rule.id.as_deref();
|
||||
if let Some(transitions) = &rule.transitions {
|
||||
for transition in transitions {
|
||||
let Some(storage_class) = &transition.storage_class else {
|
||||
continue;
|
||||
};
|
||||
let tier_name = storage_class.as_str();
|
||||
if !tier_name.is_empty() && targets.iter().any(|target| target.tier_name == tier_name) {
|
||||
return Some(TierLifecycleReference { tier_name, rule_id });
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some(noncurrent_version_transitions) = &rule.noncurrent_version_transitions {
|
||||
for transition in noncurrent_version_transitions {
|
||||
let Some(storage_class) = &transition.storage_class else {
|
||||
continue;
|
||||
};
|
||||
let tier_name = storage_class.as_str();
|
||||
if !tier_name.is_empty() && targets.iter().any(|target| target.tier_name == tier_name) {
|
||||
return Some(TierLifecycleReference { tier_name, rule_id });
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
async fn ensure_no_authoritative_persisted_references<S>(
|
||||
api: Arc<S>,
|
||||
targets: &[TierMutationIntentTarget],
|
||||
@@ -920,6 +989,15 @@ fn tier_reference_proof_persisted_in_use_error(tier_name: &str, object: &str) ->
|
||||
err
|
||||
}
|
||||
|
||||
fn tier_reference_proof_lifecycle_in_use_error(tier_name: &str, bucket: &str, rule_id: Option<&str>) -> AdminError {
|
||||
let mut err = ERR_TIER_BACKEND_IN_USE.clone();
|
||||
err.message = match rule_id {
|
||||
Some(rule_id) => format!("Remote tier {tier_name} is still referenced by lifecycle rule {rule_id} in bucket {bucket}"),
|
||||
None => format!("Remote tier {tier_name} is still referenced by a lifecycle rule in bucket {bucket}"),
|
||||
};
|
||||
err
|
||||
}
|
||||
|
||||
fn tier_reference_proof_admin_error(err: impl std::fmt::Display) -> AdminError {
|
||||
let mut admin_err = ERR_TIER_INVALID_CONFIG.clone();
|
||||
admin_err.message = format!("Remote tier reference proof failed: {err}");
|
||||
@@ -2475,6 +2553,24 @@ impl TierConfigMgr {
|
||||
})?;
|
||||
}
|
||||
|
||||
// The Azure warm backend goes through the same S3-compatible TransitionClient as every
|
||||
// other provider (backlog#2055): it has no Azure Blob-native client and no Azure AD
|
||||
// dependency, so `storage_class` and `sp_auth` cannot be honored today even though the
|
||||
// config type carries them. Reject them explicitly here instead of silently accepting
|
||||
// and then dropping them at the WarmBackendAzure construction boundary.
|
||||
if matches!(&tier_config.tier_type, TierType::Azure)
|
||||
&& let Some(azure) = tier_config.azure.as_ref()
|
||||
{
|
||||
let sp_auth_set = !azure.sp_auth.tenant_id.is_empty()
|
||||
|| !azure.sp_auth.client_id.is_empty()
|
||||
|| !azure.sp_auth.client_secret.is_empty();
|
||||
if !azure.storage_class.is_empty() || sp_auth_set {
|
||||
let mut err = ERR_TIER_INVALID_CONFIG.clone();
|
||||
err.message = "Azure remote tiers do not support storageClass or spAuth yet; leave both unset".to_string();
|
||||
return Err(err);
|
||||
}
|
||||
}
|
||||
|
||||
let d = new_warm_backend(&tier_config, true).await?;
|
||||
|
||||
if !force {
|
||||
@@ -3231,6 +3327,7 @@ impl TierConfigMgr {
|
||||
let _config_lock = config_lock;
|
||||
let _update = update;
|
||||
let mutation_kind = mutation.intent_kind();
|
||||
let check_lifecycle_references = mutation.checks_lifecycle_references();
|
||||
if version.is_none() && !candidate.tiers.is_empty() && mutation_kind != TierMutationIntentKind::Add {
|
||||
return Err(TierConfigUpdateError::Load(io::Error::other(
|
||||
"tier configuration mutation requires an existing config ETag",
|
||||
@@ -3269,7 +3366,7 @@ impl TierConfigMgr {
|
||||
let affected_targets =
|
||||
build_tier_mutation_affected_targets(mutation_kind, proof_targets, ¤t_for_targets, &candidate)
|
||||
.map_err(TierConfigUpdateError::Publish)?;
|
||||
ensure_no_authoritative_tier_object_references(api.clone(), &affected_targets)
|
||||
ensure_no_authoritative_tier_object_references(api.clone(), &affected_targets, check_lifecycle_references)
|
||||
.await
|
||||
.map_err(TierConfigUpdateError::Publish)?;
|
||||
let coordinator_intent =
|
||||
@@ -5308,6 +5405,10 @@ mod tests {
|
||||
endpoints::{Endpoints, PoolEndpoints, SetupType},
|
||||
};
|
||||
use crate::services::tier::tier_mutation_intent::TIER_MUTATION_INTENT_RECORD_PREFIX;
|
||||
use s3s::dto::{
|
||||
BucketLifecycleConfiguration, ExpirationStatus, LifecycleRule, NoncurrentVersionTransition, Transition,
|
||||
TransitionStorageClass,
|
||||
};
|
||||
|
||||
struct SetupTypeGuard {
|
||||
previous: SetupType,
|
||||
@@ -6093,6 +6194,13 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl TierReferenceProofStore for LockingTierConfigStore {
|
||||
async fn lifecycle_config_for_reference_proof(&self, _bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
|
||||
Ok(None)
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn tier_config_update_path_acquires_meta_namespace_sidecar_lock_before_save() {
|
||||
let manager = TierConfigMgr::new();
|
||||
@@ -6143,6 +6251,17 @@ mod tests {
|
||||
assert!(noop_targets.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn forced_tier_mutations_skip_only_lifecycle_reference_checks() {
|
||||
assert!(!TierCandidateMutation::Add(build_rustfs_tier("COLD-A"), true).checks_lifecycle_references());
|
||||
assert!(TierCandidateMutation::Add(build_rustfs_tier("COLD-A"), false).checks_lifecycle_references());
|
||||
assert!(!TierCandidateMutation::Remove("COLD-A".to_string(), true).checks_lifecycle_references());
|
||||
assert!(TierCandidateMutation::Remove("COLD-A".to_string(), false).checks_lifecycle_references());
|
||||
assert!(!TierCandidateMutation::Clear(true).checks_lifecycle_references());
|
||||
assert!(TierCandidateMutation::Clear(false).checks_lifecycle_references());
|
||||
assert!(TierCandidateMutation::Edit("COLD-A".to_string(), TierCreds::default()).checks_lifecycle_references());
|
||||
}
|
||||
|
||||
/// A fully offline `WarmBackend` used to exercise the driver-facing
|
||||
/// branches of `remove`/`verify` without touching a remote tier.
|
||||
struct MockWarmBackend {
|
||||
@@ -6303,6 +6422,57 @@ mod tests {
|
||||
assert!(mgr.tiers.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_add_rejects_azure_storage_class_before_backend_setup() {
|
||||
let mut mgr = empty_mgr();
|
||||
let mut tier = build_azure_tier("account-a");
|
||||
tier.azure.as_mut().expect("Azure payload should exist").storage_class = "HOT".to_string();
|
||||
|
||||
let err = mgr
|
||||
.add(tier, true)
|
||||
.await
|
||||
.expect_err("a non-empty Azure storageClass must be rejected before backend setup");
|
||||
assert_eq!(err.code, ERR_TIER_INVALID_CONFIG.code);
|
||||
assert!(err.message.contains("storageClass"), "{}", err.message);
|
||||
assert!(mgr.tiers.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_add_rejects_azure_partial_sp_auth_before_backend_setup() {
|
||||
// Only `tenant_id` is set: `TierAzure::is_sp_enabled()`-style "all three fields"
|
||||
// logic would miss this, so the check must reject on *any* sp_auth sub-field
|
||||
// being non-empty rather than requiring all three.
|
||||
let mut mgr = empty_mgr();
|
||||
let mut tier = build_azure_tier("account-a");
|
||||
tier.azure.as_mut().expect("Azure payload should exist").sp_auth.tenant_id = "tenant".to_string();
|
||||
|
||||
let err = mgr
|
||||
.add(tier, true)
|
||||
.await
|
||||
.expect_err("a partially-filled Azure spAuth must be rejected before backend setup");
|
||||
assert_eq!(err.code, ERR_TIER_INVALID_CONFIG.code);
|
||||
assert!(err.message.contains("spAuth"), "{}", err.message);
|
||||
assert!(mgr.tiers.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_add_does_not_reject_azure_config_without_storage_class_or_sp_auth() {
|
||||
// A plain Azure config (the common case: static access/secret key, no
|
||||
// storageClass, no spAuth) must sail past the new gate. `new_warm_backend`
|
||||
// builds the S3-compatible client lazily (no eager DNS/connect), so with
|
||||
// `force: true` (which also skips the `in_use` probe) this succeeds even
|
||||
// against a fake endpoint — the point here is only that the gate itself
|
||||
// does not fire.
|
||||
let mut mgr = empty_mgr();
|
||||
let tier = build_azure_tier("account-a");
|
||||
let tier_name = tier.name.clone();
|
||||
|
||||
mgr.add(tier, true)
|
||||
.await
|
||||
.expect("a config with no storageClass/spAuth must not trip the new gate");
|
||||
assert!(mgr.tiers.contains_key(&tier_name));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_add_rejects_reserved_names() {
|
||||
// Supersedes the former `test_add_does_not_reserve_standard_name_regression_anchor`
|
||||
@@ -11296,6 +11466,7 @@ mod tests {
|
||||
lock_manager: Arc<rustfs_lock::GlobalLockManager>,
|
||||
lock_requests: Mutex<Vec<(String, String)>>,
|
||||
listed_versions: Mutex<Vec<ObjectInfo>>,
|
||||
lifecycle_configs: Mutex<HashMap<String, BucketLifecycleConfiguration>>,
|
||||
}
|
||||
|
||||
impl Default for CasConfigStore {
|
||||
@@ -11316,6 +11487,7 @@ mod tests {
|
||||
lock_manager: Arc::new(rustfs_lock::GlobalLockManager::new()),
|
||||
lock_requests: Mutex::new(Vec::new()),
|
||||
listed_versions: Mutex::new(Vec::new()),
|
||||
lifecycle_configs: Mutex::new(HashMap::new()),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -11342,6 +11514,13 @@ mod tests {
|
||||
.push(object);
|
||||
}
|
||||
|
||||
fn add_lifecycle_config(&self, bucket: &str, config: BucketLifecycleConfiguration) {
|
||||
self.lifecycle_configs
|
||||
.lock()
|
||||
.expect("tier reference fixture should not poison")
|
||||
.insert(bucket.to_string(), config);
|
||||
}
|
||||
|
||||
async fn insert_config_object(&self, object: String, data: Vec<u8>) {
|
||||
self.objects
|
||||
.lock()
|
||||
@@ -11917,9 +12096,46 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl TierReferenceProofStore for CasConfigStore {
|
||||
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
|
||||
Ok(self
|
||||
.lifecycle_configs
|
||||
.lock()
|
||||
.expect("tier reference fixture should not poison")
|
||||
.get(bucket)
|
||||
.cloned())
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn remove_and_clear_full_update_paths_preserve_force() {
|
||||
let lifecycle_config = BucketLifecycleConfiguration {
|
||||
expiry_updated_at: None,
|
||||
rules: vec![LifecycleRule {
|
||||
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
|
||||
expiration: None,
|
||||
abort_incomplete_multipart_upload: None,
|
||||
del_marker_expiration: None,
|
||||
filter: None,
|
||||
id: Some("force-remove".to_string()),
|
||||
noncurrent_version_expiration: None,
|
||||
noncurrent_version_transitions: None,
|
||||
prefix: None,
|
||||
transitions: Some(vec![Transition {
|
||||
days: Some(1),
|
||||
date: None,
|
||||
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
|
||||
}]),
|
||||
}],
|
||||
};
|
||||
let remove_store = Arc::new(CasConfigStore::default());
|
||||
remove_store.add_listed_version(ObjectInfo {
|
||||
bucket: "photos".to_string(),
|
||||
name: "safe.txt".to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
remove_store.add_lifecycle_config("photos", lifecycle_config.clone());
|
||||
let mut persisted = empty_mgr();
|
||||
persisted.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
|
||||
persisted
|
||||
@@ -11946,6 +12162,12 @@ mod tests {
|
||||
);
|
||||
|
||||
let clear_store = Arc::new(CasConfigStore::default());
|
||||
clear_store.add_listed_version(ObjectInfo {
|
||||
bucket: "photos".to_string(),
|
||||
name: "safe.txt".to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
clear_store.add_lifecycle_config("photos", lifecycle_config);
|
||||
persisted
|
||||
.save_tiering_config_if_current(clear_store.clone(), None)
|
||||
.await
|
||||
@@ -12073,7 +12295,7 @@ mod tests {
|
||||
"COLD-A",
|
||||
Some(replacement_identity),
|
||||
));
|
||||
ensure_no_authoritative_tier_object_references(store.clone(), std::slice::from_ref(&target))
|
||||
ensure_no_authoritative_tier_object_references(store.clone(), std::slice::from_ref(&target), true)
|
||||
.await
|
||||
.expect("references already written with the new destination identity should not block rebind");
|
||||
|
||||
@@ -12083,13 +12305,103 @@ mod tests {
|
||||
"COLD-A",
|
||||
Some(current_identity),
|
||||
));
|
||||
let err = ensure_no_authoritative_tier_object_references(store, &[target])
|
||||
let err = ensure_no_authoritative_tier_object_references(store, &[target], true)
|
||||
.await
|
||||
.expect_err("references to the old destination identity must block rebind");
|
||||
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
|
||||
assert!(err.message.contains("photos/2026/old-destination.jpg"), "{}", err.message);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn zero_reference_proof_blocks_lifecycle_transition_references() {
|
||||
let current = build_rustfs_tier("COLD-A");
|
||||
let current_identity = tier_backend_identity(¤t).expect("current identity should encode");
|
||||
let target = TierMutationIntentTarget {
|
||||
tier_name: "COLD-A".to_string(),
|
||||
old_backend_identity: Some(current_identity),
|
||||
new_backend_identity: None,
|
||||
};
|
||||
let config = BucketLifecycleConfiguration {
|
||||
expiry_updated_at: None,
|
||||
rules: vec![LifecycleRule {
|
||||
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
|
||||
expiration: None,
|
||||
abort_incomplete_multipart_upload: None,
|
||||
del_marker_expiration: None,
|
||||
filter: None,
|
||||
id: Some("move-current".to_string()),
|
||||
noncurrent_version_expiration: None,
|
||||
noncurrent_version_transitions: None,
|
||||
prefix: None,
|
||||
transitions: Some(vec![Transition {
|
||||
days: Some(1),
|
||||
date: None,
|
||||
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
|
||||
}]),
|
||||
}],
|
||||
};
|
||||
let store = Arc::new(CasConfigStore::default());
|
||||
store.add_listed_version(ObjectInfo {
|
||||
bucket: "photos".to_string(),
|
||||
name: "safe.txt".to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
store.add_lifecycle_config("photos", config);
|
||||
|
||||
let err = ensure_no_authoritative_tier_object_references(store, &[target], true)
|
||||
.await
|
||||
.expect_err("lifecycle rule should block deletion");
|
||||
|
||||
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
|
||||
assert!(err.message.contains("move-current"), "{}", err.message);
|
||||
assert!(err.message.contains("photos"), "{}", err.message);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn zero_reference_proof_blocks_lifecycle_noncurrent_transition_references() {
|
||||
let current = build_rustfs_tier("COLD-A");
|
||||
let current_identity = tier_backend_identity(¤t).expect("current identity should encode");
|
||||
let target = TierMutationIntentTarget {
|
||||
tier_name: "COLD-A".to_string(),
|
||||
old_backend_identity: Some(current_identity),
|
||||
new_backend_identity: None,
|
||||
};
|
||||
let config = BucketLifecycleConfiguration {
|
||||
expiry_updated_at: None,
|
||||
rules: vec![LifecycleRule {
|
||||
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
|
||||
expiration: None,
|
||||
abort_incomplete_multipart_upload: None,
|
||||
del_marker_expiration: None,
|
||||
filter: None,
|
||||
id: Some("move-noncurrent".to_string()),
|
||||
noncurrent_version_expiration: None,
|
||||
noncurrent_version_transitions: Some(vec![NoncurrentVersionTransition {
|
||||
noncurrent_days: Some(1),
|
||||
newer_noncurrent_versions: None,
|
||||
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
|
||||
}]),
|
||||
prefix: None,
|
||||
transitions: None,
|
||||
}],
|
||||
};
|
||||
let store = Arc::new(CasConfigStore::default());
|
||||
store.add_listed_version(ObjectInfo {
|
||||
bucket: "photos".to_string(),
|
||||
name: "safe.txt".to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
store.add_lifecycle_config("photos", config);
|
||||
|
||||
let err = ensure_no_authoritative_tier_object_references(store, &[target], true)
|
||||
.await
|
||||
.expect_err("lifecycle rule should block deletion");
|
||||
|
||||
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
|
||||
assert!(err.message.contains("move-noncurrent"), "{}", err.message);
|
||||
assert!(err.message.contains("photos"), "{}", err.message);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn zero_reference_proof_blocks_persisted_journal_transaction_and_free_version_references() {
|
||||
let current = build_rustfs_tier("COLD-A");
|
||||
@@ -12120,7 +12432,7 @@ mod tests {
|
||||
.expect("journal record should encode"),
|
||||
)
|
||||
.await;
|
||||
let err = ensure_no_authoritative_tier_object_references(journal_store, std::slice::from_ref(&target))
|
||||
let err = ensure_no_authoritative_tier_object_references(journal_store, std::slice::from_ref(&target), true)
|
||||
.await
|
||||
.expect_err("unfinished delete journal for the old backend must block rebind");
|
||||
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
|
||||
@@ -12158,7 +12470,7 @@ mod tests {
|
||||
transaction.encode().expect("transaction record should encode"),
|
||||
)
|
||||
.await;
|
||||
let err = ensure_no_authoritative_tier_object_references(transaction_store, std::slice::from_ref(&target))
|
||||
let err = ensure_no_authoritative_tier_object_references(transaction_store, std::slice::from_ref(&target), true)
|
||||
.await
|
||||
.expect_err("unfinished transition transaction for the old backend must block rebind");
|
||||
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
|
||||
@@ -12168,11 +12480,11 @@ mod tests {
|
||||
let mut free_version = transitioned_tier_object("photos", "2026/free-version.jpg", "COLD-A", Some(current_identity));
|
||||
free_version.transitioned_object.status = "pending".to_string();
|
||||
free_version.transitioned_object.free_version = true;
|
||||
ensure_no_authoritative_tier_object_references(free_version_store.clone(), std::slice::from_ref(&target))
|
||||
ensure_no_authoritative_tier_object_references(free_version_store.clone(), std::slice::from_ref(&target), true)
|
||||
.await
|
||||
.expect("empty free-version fixture should permit rebind");
|
||||
free_version_store.add_listed_version(free_version);
|
||||
let err = ensure_no_authoritative_tier_object_references(free_version_store, &[target])
|
||||
let err = ensure_no_authoritative_tier_object_references(free_version_store, &[target], true)
|
||||
.await
|
||||
.expect_err("recoverable free version for the old backend must block rebind");
|
||||
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
|
||||
@@ -12198,7 +12510,7 @@ mod tests {
|
||||
}
|
||||
store.omit_truncated_reference_marker();
|
||||
|
||||
let err = ensure_no_authoritative_tier_object_references(store, &[target])
|
||||
let err = ensure_no_authoritative_tier_object_references(store, &[target], true)
|
||||
.await
|
||||
.expect_err("truncated authoritative reference scan without a marker must fail closed");
|
||||
assert_eq!(err.code, ERR_TIER_INVALID_CONFIG.code);
|
||||
|
||||
@@ -646,12 +646,6 @@ pub struct TierAzure {
|
||||
pub sp_auth: ServicePrincipalAuth,
|
||||
}
|
||||
|
||||
impl TierAzure {
|
||||
pub fn is_sp_enabled(&self) -> bool {
|
||||
!self.sp_auth.tenant_id.is_empty() && !self.sp_auth.client_id.is_empty() && !self.sp_auth.client_secret.is_empty()
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
fn AzureServicePrincipal(tenantID, clientID, clientSecret string) func(az *TierAzure) error {
|
||||
return func(az *TierAzure) error {
|
||||
|
||||
@@ -36,11 +36,14 @@ use crate::services::tier::{
|
||||
};
|
||||
use bytes::Bytes;
|
||||
use http::StatusCode;
|
||||
use rustfs_s3_client::credentials::{Credentials, SignatureType, Static, Value};
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, Options, TransitionClient, TransitionCore};
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::{AdvancedPutOptions, PutObjectOptions},
|
||||
transition_api::{ReadCloser, ReaderImpl},
|
||||
};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
use rustfs_utils::http::headers::{
|
||||
CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE, EXPIRES, HeaderExt as _,
|
||||
};
|
||||
@@ -50,6 +53,7 @@ use s3s::header::{
|
||||
X_AMZ_STORAGE_CLASS,
|
||||
};
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use time::OffsetDateTime;
|
||||
use time::format_description::well_known::{Rfc2822, Rfc3339};
|
||||
use tracing::{info, warn};
|
||||
@@ -58,6 +62,11 @@ pub type WarmBackendImpl = Box<dyn WarmBackend + Send + Sync + 'static>;
|
||||
|
||||
const PROBE_OBJECT: &str = "probeobject";
|
||||
|
||||
/// Largest object the S3-compatible warm backends accept for a multipart put.
|
||||
pub(crate) const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
/// Part-count ceiling S3-compatible services impose on a multipart upload.
|
||||
pub(crate) const MAX_PARTS_COUNT: i64 = 10000;
|
||||
|
||||
#[derive(Default)]
|
||||
pub struct WarmBackendGetOpts {
|
||||
pub start_offset: i64,
|
||||
@@ -222,6 +231,121 @@ pub fn build_transition_put_options(storage_class: String, mut metadata: HashMap
|
||||
opts
|
||||
}
|
||||
|
||||
/// Connection parameters every S3-compatible warm backend provider supplies.
|
||||
///
|
||||
/// The Aliyun, Azure, Huaweicloud, Tencent, MinIO, R2, and RustFS backends all
|
||||
/// wrap [`WarmBackendS3`] around a statically-credentialed [`TransitionClient`]
|
||||
/// built from exactly these values. `bucket_lookup` is a parameter rather than a
|
||||
/// constant because the providers split into two families: Aliyun, Azure,
|
||||
/// Huaweicloud, and Tencent pin [`BucketLookupType::BucketLookupDNS`], while
|
||||
/// MinIO, R2, and RustFS leave it at [`BucketLookupType::BucketLookupAuto`].
|
||||
pub(crate) struct S3CompatibleWarmBackendParams<'a> {
|
||||
pub endpoint: &'a str,
|
||||
pub access_key: &'a str,
|
||||
pub secret_key: &'a str,
|
||||
pub bucket: &'a str,
|
||||
pub prefix: &'a str,
|
||||
pub region: &'a str,
|
||||
pub bucket_lookup: BucketLookupType,
|
||||
/// Tag handed to [`TransitionClient::new`] so per-provider client behavior
|
||||
/// and metrics stay attributable.
|
||||
pub provider_tag: &'a str,
|
||||
/// SSRF guard run against the parsed endpoint once it's known to have a
|
||||
/// host. Almost every provider passes [`rustfs_utils::egress::validate_outbound_url`]
|
||||
/// unchanged; RustFS passes its own wrapper that adds a debug-only,
|
||||
/// env-gated loopback exception for its e2e tier tests (see
|
||||
/// rustfs/rustfs#6773) — the shared constructor stays the single call
|
||||
/// site either way, so no provider can silently end up unvalidated.
|
||||
pub validate_endpoint: fn(&url::Url) -> Result<(), rustfs_utils::egress::OutboundUrlError>,
|
||||
}
|
||||
|
||||
/// Build the [`WarmBackendS3`] shared by the S3-compatible warm backend providers.
|
||||
///
|
||||
/// Credential, bucket, and endpoint validation run in this order because the
|
||||
/// existing provider constructors report the first failure they hit, and their
|
||||
/// error texts are user-visible through the tier admin API.
|
||||
pub(crate) async fn new_s3_compatible_warm_backend(
|
||||
params: S3CompatibleWarmBackendParams<'_>,
|
||||
) -> Result<WarmBackendS3, std::io::Error> {
|
||||
if params.access_key.is_empty() || params.secret_key.is_empty() {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if params.bucket.is_empty() {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
let u = match url::Url::parse(params.endpoint) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
return Err(std::io::Error::other(e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: params.access_key.to_string(),
|
||||
secret_access_key: params.secret_key.to_string(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: params.region.to_string(),
|
||||
bucket_lookup: params.bucket_lookup,
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
|
||||
// Runs after the host-presence check above (not immediately after Url::parse) so a
|
||||
// host-less endpoint still reports this constructor's own "missing host" text instead of
|
||||
// validate_endpoint's differently-worded rejection for the same input.
|
||||
(params.validate_endpoint)(&u).map_err(|err| std::io::Error::other(format!("tier endpoint is not allowed: {err}")))?;
|
||||
let client =
|
||||
TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, params.provider_tag).await?;
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: params.bucket.to_string(),
|
||||
prefix: params.prefix.strip_suffix("/").unwrap_or(params.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Round the multipart part size up to a whole multiple of `min_part_size` that
|
||||
/// keeps the upload within [`MAX_PARTS_COUNT`] parts.
|
||||
///
|
||||
/// `object_size == -1` means "length unknown", so the caller is charged the
|
||||
/// worst case of a full [`MAX_MULTIPART_PUT_OBJECT_SIZE`] object.
|
||||
pub(crate) fn optimal_part_size(object_size: i64, min_part_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
|
||||
let configured_part_size = min_part_size;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(min_part_size);
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
pub async fn check_warm_backend(w: Option<&WarmBackendImpl>) -> Result<(), AdminError> {
|
||||
let w = w.ok_or_else(|| ERR_TIER_NOT_FOUND.clone())?;
|
||||
w.validate().await.map_err(|_| ERR_TIER_INVALID_CONFIG.clone())?;
|
||||
@@ -803,6 +927,194 @@ mod tests {
|
||||
assert_eq!(err.code, ERR_TIER_INVALID_CONFIG.code);
|
||||
}
|
||||
|
||||
/// Every S3-compatible provider file pins this same floor today.
|
||||
const PROVIDER_MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
fn s3_compatible_params(endpoint: &str) -> S3CompatibleWarmBackendParams<'_> {
|
||||
S3CompatibleWarmBackendParams {
|
||||
endpoint,
|
||||
access_key: "access",
|
||||
secret_key: "secret",
|
||||
bucket: "tier-bucket",
|
||||
prefix: "archive",
|
||||
region: "us-east-1",
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
provider_tag: "aliyun",
|
||||
validate_endpoint: validate_outbound_url,
|
||||
}
|
||||
}
|
||||
|
||||
/// `WarmBackendS3` has no `Debug`, so `Result::expect_err` is unavailable.
|
||||
async fn init_error(params: S3CompatibleWarmBackendParams<'_>, must_fail_because: &str) -> std::io::Error {
|
||||
match new_s3_compatible_warm_backend(params).await {
|
||||
Ok(_) => panic!("{must_fail_because}"),
|
||||
Err(err) => err,
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_rejects_missing_credentials_before_parsing_the_endpoint() {
|
||||
let mut params = s3_compatible_params("://not-a-url");
|
||||
params.access_key = "";
|
||||
let err = init_error(params, "an empty access key must be rejected").await;
|
||||
assert_eq!(err.to_string(), "both access and secret keys are required");
|
||||
|
||||
let mut params = s3_compatible_params("://not-a-url");
|
||||
params.secret_key = "";
|
||||
let err = init_error(params, "an empty secret key must be rejected").await;
|
||||
assert_eq!(err.to_string(), "both access and secret keys are required");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_rejects_an_empty_bucket_before_parsing_the_endpoint() {
|
||||
let mut params = s3_compatible_params("://not-a-url");
|
||||
params.bucket = "";
|
||||
|
||||
let err = init_error(params, "an empty bucket must be rejected").await;
|
||||
|
||||
assert_eq!(err.to_string(), "no bucket name was provided");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_rejects_an_unparsable_endpoint() {
|
||||
let err = init_error(s3_compatible_params("://not-a-url"), "an endpoint that is not a URL must be rejected").await;
|
||||
|
||||
assert_eq!(err.to_string(), url::ParseError::RelativeUrlWithoutBase.to_string());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_rejects_an_endpoint_without_a_host() {
|
||||
let err = init_error(s3_compatible_params("rustfs://"), "an endpoint without a host must be rejected").await;
|
||||
|
||||
assert_eq!(err.to_string(), "Invalid endpoint URL: missing host");
|
||||
}
|
||||
|
||||
/// Every migrated provider that uses `validate_outbound_url` directly (all
|
||||
/// but RustFS, which injects its own debug-only, env-gated wrapper — see
|
||||
/// rustfs/rustfs#6773) goes through this one construction path, so the
|
||||
/// SSRF guard only needs to be pinned here rather than once per provider
|
||||
/// file (see backlog#2040's migrate steps and rustfs/rustfs#6764).
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_rejects_a_loopback_endpoint_before_any_network_setup() {
|
||||
let err = init_error(s3_compatible_params("https://127.0.0.1:9000"), "a loopback endpoint must be rejected").await;
|
||||
|
||||
assert!(err.to_string().contains("not allowed"), "unexpected error: {err}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_carries_provider_options_to_the_transition_client() {
|
||||
let backend = new_s3_compatible_warm_backend(s3_compatible_params("http://tier.example.com:9000"))
|
||||
.await
|
||||
.expect("a well-formed S3-compatible tier config should initialize offline");
|
||||
|
||||
assert_eq!(backend.bucket, "tier-bucket");
|
||||
assert_eq!(backend.prefix, "archive");
|
||||
assert_eq!(backend.storage_class, "");
|
||||
assert!(!backend.client.secure);
|
||||
assert_eq!(backend.client.endpoint_url.scheme(), "http");
|
||||
assert_eq!(backend.client.endpoint_url.host_str(), Some("tier.example.com"));
|
||||
assert_eq!(backend.client.endpoint_url.port(), Some(9000));
|
||||
assert_eq!(backend.client.region, "us-east-1");
|
||||
assert_eq!(backend.client.lookup, BucketLookupType::BucketLookupDNS);
|
||||
// The provider constructors all request `trailing_headers: true`, but
|
||||
// `TransitionClient` gates the feature on an explicitly overridden SigV4
|
||||
// signer, which none of them set. Pin the resulting `false` so migrating
|
||||
// a provider onto this constructor cannot silently flip wire behavior.
|
||||
assert!(!backend.client.trailing_header_support);
|
||||
assert_eq!(backend.client.tier_type, "aliyun");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_derives_tls_and_the_default_port_from_the_scheme() {
|
||||
let secure = new_s3_compatible_warm_backend(s3_compatible_params("https://tier.example.com"))
|
||||
.await
|
||||
.expect("an https endpoint should initialize offline");
|
||||
assert!(secure.client.secure);
|
||||
assert_eq!(secure.client.endpoint_url.scheme(), "https");
|
||||
assert_eq!(secure.client.endpoint_url.port_or_known_default(), Some(443));
|
||||
|
||||
let insecure = new_s3_compatible_warm_backend(s3_compatible_params("http://tier.example.com"))
|
||||
.await
|
||||
.expect("an http endpoint should initialize offline");
|
||||
assert!(!insecure.client.secure);
|
||||
assert_eq!(insecure.client.endpoint_url.scheme(), "http");
|
||||
assert_eq!(insecure.client.endpoint_url.port_or_known_default(), Some(80));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_strips_only_a_trailing_prefix_separator() {
|
||||
let mut params = s3_compatible_params("http://tier.example.com:9000");
|
||||
params.prefix = "archive/";
|
||||
let trimmed = new_s3_compatible_warm_backend(params)
|
||||
.await
|
||||
.expect("a prefix with a trailing separator should initialize offline");
|
||||
assert_eq!(trimmed.prefix, "archive");
|
||||
|
||||
let mut params = s3_compatible_params("http://tier.example.com:9000");
|
||||
params.prefix = "archive/nested";
|
||||
let untouched = new_s3_compatible_warm_backend(params)
|
||||
.await
|
||||
.expect("a nested prefix should initialize offline");
|
||||
assert_eq!(untouched.prefix, "archive/nested");
|
||||
|
||||
let mut params = s3_compatible_params("http://tier.example.com:9000");
|
||||
params.prefix = "";
|
||||
let empty = new_s3_compatible_warm_backend(params)
|
||||
.await
|
||||
.expect("an empty prefix should initialize offline");
|
||||
assert_eq!(empty.prefix, "");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_honors_the_auto_bucket_lookup_family() {
|
||||
let mut params = s3_compatible_params("http://tier.example.com:9000");
|
||||
params.bucket_lookup = BucketLookupType::BucketLookupAuto;
|
||||
params.provider_tag = "minio";
|
||||
|
||||
let backend = new_s3_compatible_warm_backend(params)
|
||||
.await
|
||||
.expect("the auto-lookup provider family should initialize offline");
|
||||
|
||||
assert_eq!(backend.client.lookup, BucketLookupType::BucketLookupAuto);
|
||||
assert_eq!(backend.client.tier_type, "minio");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn optimal_part_size_charges_an_unknown_length_the_multipart_ceiling() {
|
||||
let unknown = optimal_part_size(-1, PROVIDER_MIN_PART_SIZE).expect("an unknown length must be accepted");
|
||||
let ceiling =
|
||||
optimal_part_size(MAX_MULTIPART_PUT_OBJECT_SIZE, PROVIDER_MIN_PART_SIZE).expect("the exact ceiling must be accepted");
|
||||
|
||||
assert_eq!(unknown, ceiling);
|
||||
assert_eq!(unknown, 5 * PROVIDER_MIN_PART_SIZE);
|
||||
assert!(unknown * MAX_PARTS_COUNT >= MAX_MULTIPART_PUT_OBJECT_SIZE);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn optimal_part_size_rejects_an_object_above_the_multipart_ceiling() {
|
||||
let err = optimal_part_size(MAX_MULTIPART_PUT_OBJECT_SIZE + 1, PROVIDER_MIN_PART_SIZE)
|
||||
.expect_err("an object past the multipart ceiling must fail closed");
|
||||
|
||||
assert_eq!(err.to_string(), "entity too large");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn optimal_part_size_never_returns_less_than_one_part() {
|
||||
assert_eq!(
|
||||
optimal_part_size(0, PROVIDER_MIN_PART_SIZE).expect("a zero-length object must be accepted"),
|
||||
PROVIDER_MIN_PART_SIZE
|
||||
);
|
||||
assert_eq!(
|
||||
optimal_part_size(1024, PROVIDER_MIN_PART_SIZE).expect("a tiny object must be accepted"),
|
||||
PROVIDER_MIN_PART_SIZE
|
||||
);
|
||||
assert_eq!(
|
||||
optimal_part_size(PROVIDER_MIN_PART_SIZE, PROVIDER_MIN_PART_SIZE)
|
||||
.expect("an object of exactly one part must be accepted"),
|
||||
PROVIDER_MIN_PART_SIZE
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_transition_put_options_preserves_content_headers() {
|
||||
let mut metadata = HashMap::new();
|
||||
|
||||
@@ -19,76 +19,38 @@
|
||||
#![allow(clippy::all)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::services::tier::{
|
||||
tier_config::TierAliyun,
|
||||
warm_backend::{WarmBackend, WarmBackendGetOpts, build_transition_put_options},
|
||||
warm_backend::{
|
||||
S3CompatibleWarmBackendParams, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
|
||||
new_s3_compatible_warm_backend, optimal_part_size,
|
||||
},
|
||||
warm_backend_s3::WarmBackendS3,
|
||||
};
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::PutObjectOptions,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
transition_api::{BucketLookupType, Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
|
||||
};
|
||||
use tracing::warn;
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
|
||||
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
const MAX_PARTS_COUNT: i64 = 10000;
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
pub struct WarmBackendAliyun(WarmBackendS3);
|
||||
|
||||
impl WarmBackendAliyun {
|
||||
pub async fn new(conf: &TierAliyun, tier: &str) -> Result<Self, std::io::Error> {
|
||||
if conf.access_key == "" || conf.secret_key == "" {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if conf.bucket == "" {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
let u = match url::Url::parse(&conf.endpoint) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
return Err(std::io::Error::other(e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: conf.access_key.clone(),
|
||||
secret_access_key: conf.secret_key.clone(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: conf.region.clone(),
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
|
||||
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "aliyun").await?;
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(Self(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
}))
|
||||
Ok(Self(
|
||||
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
|
||||
endpoint: &conf.endpoint,
|
||||
access_key: &conf.access_key,
|
||||
secret_key: &conf.secret_key,
|
||||
bucket: &conf.bucket,
|
||||
prefix: &conf.prefix,
|
||||
region: &conf.region,
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
provider_tag: "aliyun",
|
||||
validate_endpoint: validate_outbound_url,
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -101,7 +63,7 @@ impl WarmBackend for WarmBackendAliyun {
|
||||
length: i64,
|
||||
meta: HashMap<String, String>,
|
||||
) -> Result<String, std::io::Error> {
|
||||
let part_size = optimal_part_size(length)?;
|
||||
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
|
||||
let client = self.0.client.clone();
|
||||
let res = client
|
||||
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
|
||||
@@ -132,23 +94,29 @@ impl WarmBackend for WarmBackendAliyun {
|
||||
}
|
||||
}
|
||||
|
||||
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::services::tier::tier_config::TierAliyun;
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
/// The SSRF guard itself is exercised once, generically, in
|
||||
/// `warm_backend::tests` (see backlog#2040/backlog#2041 and
|
||||
/// rustfs/rustfs#6764) — this test only pins that this provider's
|
||||
/// production constructor really is wired through that shared path.
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_network_setup() {
|
||||
let conf = TierAliyun {
|
||||
endpoint: "https://127.0.0.1:9000".to_string(),
|
||||
bucket: "tier-bucket".to_string(),
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
region: "us-east-1".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let configured_part_size = MIN_PART_SIZE;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(MIN_PART_SIZE);
|
||||
match WarmBackendAliyun::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed")),
|
||||
}
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
@@ -19,76 +19,38 @@
|
||||
#![allow(clippy::all)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::services::tier::{
|
||||
tier_config::TierAzure,
|
||||
warm_backend::{WarmBackend, WarmBackendGetOpts, build_transition_put_options},
|
||||
warm_backend::{
|
||||
S3CompatibleWarmBackendParams, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
|
||||
new_s3_compatible_warm_backend, optimal_part_size,
|
||||
},
|
||||
warm_backend_s3::WarmBackendS3,
|
||||
};
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::PutObjectOptions,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
transition_api::{BucketLookupType, Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
|
||||
};
|
||||
use tracing::warn;
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
|
||||
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
const MAX_PARTS_COUNT: i64 = 10000;
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
pub struct WarmBackendAzure(WarmBackendS3);
|
||||
|
||||
impl WarmBackendAzure {
|
||||
pub async fn new(conf: &TierAzure, tier: &str) -> Result<Self, std::io::Error> {
|
||||
if conf.access_key == "" || conf.secret_key == "" {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if conf.bucket == "" {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
let u = match url::Url::parse(&conf.endpoint) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
return Err(std::io::Error::other(e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: conf.access_key.clone(),
|
||||
secret_access_key: conf.secret_key.clone(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: conf.region.clone(),
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
|
||||
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "azure").await?;
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(Self(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
}))
|
||||
Ok(Self(
|
||||
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
|
||||
endpoint: &conf.endpoint,
|
||||
access_key: &conf.access_key,
|
||||
secret_key: &conf.secret_key,
|
||||
bucket: &conf.bucket,
|
||||
prefix: &conf.prefix,
|
||||
region: &conf.region,
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
provider_tag: "azure",
|
||||
validate_endpoint: validate_outbound_url,
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -101,7 +63,7 @@ impl WarmBackend for WarmBackendAzure {
|
||||
length: i64,
|
||||
meta: HashMap<String, String>,
|
||||
) -> Result<String, std::io::Error> {
|
||||
let part_size = optimal_part_size(length)?;
|
||||
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
|
||||
let client = self.0.client.clone();
|
||||
let res = client
|
||||
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
|
||||
@@ -132,23 +94,29 @@ impl WarmBackend for WarmBackendAzure {
|
||||
}
|
||||
}
|
||||
|
||||
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::services::tier::tier_config::TierAzure;
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
/// The SSRF guard itself is exercised once, generically, in
|
||||
/// `warm_backend::tests` (see backlog#2040/backlog#2041 and
|
||||
/// rustfs/rustfs#6764) — this test only pins that this provider's
|
||||
/// production constructor really is wired through that shared path.
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_network_setup() {
|
||||
let conf = TierAzure {
|
||||
endpoint: "https://127.0.0.1:9000".to_string(),
|
||||
bucket: "tier-bucket".to_string(),
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
region: "us-east-1".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let configured_part_size = MIN_PART_SIZE;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(MIN_PART_SIZE);
|
||||
match WarmBackendAzure::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed")),
|
||||
}
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
@@ -39,6 +39,7 @@ use rustfs_s3_client::{
|
||||
api_put_object::PutObjectOptions,
|
||||
transition_api::{Options, ReadCloser, ReaderImpl},
|
||||
};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
use tracing::warn;
|
||||
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
@@ -73,6 +74,12 @@ impl WarmBackendGCS {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
if !conf.endpoint.is_empty() {
|
||||
let endpoint_url = url::Url::parse(&conf.endpoint).map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
validate_outbound_url(&endpoint_url)
|
||||
.map_err(|err| std::io::Error::other(format!("tier endpoint is not allowed: {err}")))?;
|
||||
}
|
||||
|
||||
let authorized_user = serde_json::from_str(&conf.creds)?;
|
||||
let credentials = Builder::new(authorized_user)
|
||||
//.with_retry_policy(AlwaysRetry.with_attempt_limit(3))
|
||||
@@ -211,7 +218,9 @@ impl WarmBackend for WarmBackendGCS {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::WarmBackendGCS;
|
||||
use super::parse_generation;
|
||||
use crate::services::tier::tier_config::TierGCS;
|
||||
use std::io::ErrorKind;
|
||||
|
||||
#[test]
|
||||
@@ -231,6 +240,21 @@ mod tests {
|
||||
assert_eq!(err.kind(), ErrorKind::InvalidData, "{value}");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_credential_setup() {
|
||||
let conf = TierGCS {
|
||||
endpoint: "https://127.0.0.1:9000".to_string(),
|
||||
creds: "not-json".to_string(),
|
||||
bucket: "tier-bucket".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
match WarmBackendGCS::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed"), "unexpected error: {err}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/*fn gcs_to_object_error(err: Error, params: Vec<String>) -> Option<Error> {
|
||||
|
||||
@@ -19,77 +19,38 @@
|
||||
#![allow(clippy::all)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::services::tier::{
|
||||
tier_config::TierHuaweicloud,
|
||||
warm_backend::{WarmBackend, WarmBackendGetOpts, build_transition_put_options},
|
||||
warm_backend::{
|
||||
S3CompatibleWarmBackendParams, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
|
||||
new_s3_compatible_warm_backend, optimal_part_size,
|
||||
},
|
||||
warm_backend_s3::WarmBackendS3,
|
||||
};
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::PutObjectOptions,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
transition_api::{BucketLookupType, Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
|
||||
};
|
||||
use tracing::warn;
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
|
||||
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
const MAX_PARTS_COUNT: i64 = 10000;
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
pub struct WarmBackendHuaweicloud(WarmBackendS3);
|
||||
|
||||
impl WarmBackendHuaweicloud {
|
||||
pub async fn new(conf: &TierHuaweicloud, tier: &str) -> Result<Self, std::io::Error> {
|
||||
if conf.access_key == "" || conf.secret_key == "" {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if conf.bucket == "" {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
let u = match url::Url::parse(&conf.endpoint) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
return Err(std::io::Error::other(e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: conf.access_key.clone(),
|
||||
secret_access_key: conf.secret_key.clone(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: conf.region.clone(),
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
|
||||
let client =
|
||||
TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "huaweicloud").await?;
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(Self(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
}))
|
||||
Ok(Self(
|
||||
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
|
||||
endpoint: &conf.endpoint,
|
||||
access_key: &conf.access_key,
|
||||
secret_key: &conf.secret_key,
|
||||
bucket: &conf.bucket,
|
||||
prefix: &conf.prefix,
|
||||
region: &conf.region,
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
provider_tag: "huaweicloud",
|
||||
validate_endpoint: validate_outbound_url,
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -102,7 +63,7 @@ impl WarmBackend for WarmBackendHuaweicloud {
|
||||
length: i64,
|
||||
meta: HashMap<String, String>,
|
||||
) -> Result<String, std::io::Error> {
|
||||
let part_size = optimal_part_size(length)?;
|
||||
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
|
||||
let client = self.0.client.clone();
|
||||
let res = client
|
||||
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
|
||||
@@ -133,23 +94,29 @@ impl WarmBackend for WarmBackendHuaweicloud {
|
||||
}
|
||||
}
|
||||
|
||||
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::services::tier::tier_config::TierHuaweicloud;
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
/// The SSRF guard itself is exercised once, generically, in
|
||||
/// `warm_backend::tests` (see backlog#2040/backlog#2041 and
|
||||
/// rustfs/rustfs#6764) — this test only pins that this provider's
|
||||
/// production constructor really is wired through that shared path.
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_network_setup() {
|
||||
let conf = TierHuaweicloud {
|
||||
endpoint: "https://127.0.0.1:9000".to_string(),
|
||||
bucket: "tier-bucket".to_string(),
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
region: "us-east-1".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let configured_part_size = MIN_PART_SIZE;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(MIN_PART_SIZE);
|
||||
match WarmBackendHuaweicloud::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed")),
|
||||
}
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
@@ -19,23 +19,18 @@
|
||||
#![allow(clippy::all)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::services::tier::{
|
||||
tier_config::TierMinIO,
|
||||
warm_backend::{TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options},
|
||||
warm_backend::{
|
||||
S3CompatibleWarmBackendParams, TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
|
||||
new_s3_compatible_warm_backend, optimal_part_size,
|
||||
},
|
||||
warm_backend_s3::WarmBackendS3,
|
||||
};
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::PutObjectOptions,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
transition_api::{Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
|
||||
};
|
||||
use tracing::warn;
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
|
||||
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
const MAX_PARTS_COUNT: i64 = 10000;
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
@@ -43,51 +38,22 @@ pub struct WarmBackendMinIO(WarmBackendS3);
|
||||
|
||||
impl WarmBackendMinIO {
|
||||
pub async fn new(conf: &TierMinIO, tier: &str) -> Result<Self, std::io::Error> {
|
||||
if conf.access_key == "" || conf.secret_key == "" {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if conf.bucket == "" {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
let u = match url::Url::parse(&conf.endpoint) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
return Err(std::io::Error::other(e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: conf.access_key.clone(),
|
||||
secret_access_key: conf.secret_key.clone(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: conf.region.clone(),
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
|
||||
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "minio").await?;
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(Self(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
}))
|
||||
Ok(Self(
|
||||
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
|
||||
endpoint: &conf.endpoint,
|
||||
access_key: &conf.access_key,
|
||||
secret_key: &conf.secret_key,
|
||||
bucket: &conf.bucket,
|
||||
prefix: &conf.prefix,
|
||||
region: &conf.region,
|
||||
// MinIO tier endpoints are commonly path-style, so bucket addressing stays on
|
||||
// `BucketLookupAuto`; pinning DNS here would break those deployments.
|
||||
bucket_lookup: BucketLookupType::BucketLookupAuto,
|
||||
provider_tag: "minio",
|
||||
validate_endpoint: validate_outbound_url,
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -100,7 +66,7 @@ impl WarmBackend for WarmBackendMinIO {
|
||||
length: i64,
|
||||
meta: HashMap<String, String>,
|
||||
) -> Result<String, std::io::Error> {
|
||||
let part_size = optimal_part_size(length)?;
|
||||
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
|
||||
let client = self.0.client.clone();
|
||||
let res = client
|
||||
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
|
||||
@@ -149,23 +115,29 @@ impl crate::services::tier::warm_backend::TransitionCandidateReconciler for Warm
|
||||
}
|
||||
}
|
||||
|
||||
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::services::tier::tier_config::TierMinIO;
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
/// The SSRF guard itself is exercised once, generically, in
|
||||
/// `warm_backend::tests` (see backlog#2040/backlog#2043 and
|
||||
/// rustfs/rustfs#6764) — this test only pins that this provider's
|
||||
/// production constructor really is wired through that shared path.
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_network_setup() {
|
||||
let conf = TierMinIO {
|
||||
endpoint: "https://127.0.0.1:9000".to_string(),
|
||||
bucket: "tier-bucket".to_string(),
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
region: "us-east-1".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let configured_part_size = MIN_PART_SIZE;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(MIN_PART_SIZE);
|
||||
match WarmBackendMinIO::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed")),
|
||||
}
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
@@ -19,23 +19,18 @@
|
||||
#![allow(clippy::all)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::services::tier::{
|
||||
tier_config::TierR2,
|
||||
warm_backend::{TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options},
|
||||
warm_backend::{
|
||||
S3CompatibleWarmBackendParams, TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
|
||||
new_s3_compatible_warm_backend, optimal_part_size,
|
||||
},
|
||||
warm_backend_s3::WarmBackendS3,
|
||||
};
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::PutObjectOptions,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
transition_api::{Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
|
||||
};
|
||||
use tracing::warn;
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
|
||||
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
const MAX_PARTS_COUNT: i64 = 10000;
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
@@ -43,51 +38,22 @@ pub struct WarmBackendR2(WarmBackendS3);
|
||||
|
||||
impl WarmBackendR2 {
|
||||
pub async fn new(conf: &TierR2, tier: &str) -> Result<Self, std::io::Error> {
|
||||
if conf.access_key == "" || conf.secret_key == "" {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if conf.bucket == "" {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
let u = match url::Url::parse(&conf.endpoint) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
return Err(std::io::Error::other(e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: conf.access_key.clone(),
|
||||
secret_access_key: conf.secret_key.clone(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: conf.region.clone(),
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
|
||||
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "r2").await?;
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(Self(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
}))
|
||||
Ok(Self(
|
||||
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
|
||||
endpoint: &conf.endpoint,
|
||||
access_key: &conf.access_key,
|
||||
secret_key: &conf.secret_key,
|
||||
bucket: &conf.bucket,
|
||||
prefix: &conf.prefix,
|
||||
region: &conf.region,
|
||||
// R2 tier endpoints are commonly path-style, so bucket addressing stays on
|
||||
// `BucketLookupAuto`; pinning DNS here would break those deployments.
|
||||
bucket_lookup: BucketLookupType::BucketLookupAuto,
|
||||
provider_tag: "r2",
|
||||
validate_endpoint: validate_outbound_url,
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -100,7 +66,7 @@ impl WarmBackend for WarmBackendR2 {
|
||||
length: i64,
|
||||
meta: HashMap<String, String>,
|
||||
) -> Result<String, std::io::Error> {
|
||||
let part_size = optimal_part_size(length)?;
|
||||
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
|
||||
let client = self.0.client.clone();
|
||||
let res = client
|
||||
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
|
||||
@@ -149,23 +115,29 @@ impl crate::services::tier::warm_backend::TransitionCandidateReconciler for Warm
|
||||
}
|
||||
}
|
||||
|
||||
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::services::tier::tier_config::TierR2;
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
/// The SSRF guard itself is exercised once, generically, in
|
||||
/// `warm_backend::tests` (see backlog#2040/backlog#2043 and
|
||||
/// rustfs/rustfs#6764) — this test only pins that this provider's
|
||||
/// production constructor really is wired through that shared path.
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_network_setup() {
|
||||
let conf = TierR2 {
|
||||
endpoint: "https://127.0.0.1:9000".to_string(),
|
||||
bucket: "tier-bucket".to_string(),
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
region: "us-east-1".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let configured_part_size = MIN_PART_SIZE;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(MIN_PART_SIZE);
|
||||
match WarmBackendR2::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed")),
|
||||
}
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
@@ -19,34 +19,55 @@
|
||||
#![allow(clippy::all)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::services::tier::{
|
||||
tier_config::TierRustFS,
|
||||
warm_backend::{TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options},
|
||||
warm_backend::{
|
||||
S3CompatibleWarmBackendParams, TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
|
||||
new_s3_compatible_warm_backend, optimal_part_size,
|
||||
},
|
||||
warm_backend_s3::WarmBackendS3,
|
||||
};
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::PutObjectOptions,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
transition_api::{Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
|
||||
};
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
|
||||
use rustfs_utils::egress::{OutboundUrlError, validate_outbound_url};
|
||||
|
||||
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
const MAX_PARTS_COUNT: i64 = 10000;
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
// Debug-only opt-in for single-host test/dev setups; release builds always reject loopback.
|
||||
const ALLOW_LOOPBACK_TIER_ENDPOINT_ENV: &str = "RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT";
|
||||
|
||||
fn validate_rustfs_tier_endpoint(url: &url::Url) -> Result<(), OutboundUrlError> {
|
||||
let allow_loopback = cfg!(debug_assertions)
|
||||
&& std::env::var(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV)
|
||||
.map(|value| value == "1" || value.eq_ignore_ascii_case("true"))
|
||||
.unwrap_or(false);
|
||||
validate_rustfs_tier_endpoint_inner(url, allow_loopback)
|
||||
}
|
||||
|
||||
fn validate_rustfs_tier_endpoint_inner(url: &url::Url, allow_loopback: bool) -> Result<(), OutboundUrlError> {
|
||||
match validate_outbound_url(url) {
|
||||
Err(OutboundUrlError::ForbiddenHost {
|
||||
reason: "loopback address" | "loopback host",
|
||||
..
|
||||
}) if allow_loopback => Ok(()),
|
||||
result => result,
|
||||
}
|
||||
}
|
||||
|
||||
pub struct WarmBackendRustFS(WarmBackendS3);
|
||||
|
||||
impl WarmBackendRustFS {
|
||||
pub async fn new(conf: &TierRustFS, tier: &str) -> Result<Self, std::io::Error> {
|
||||
if conf.access_key == "" || conf.secret_key == "" {
|
||||
// This provider reports endpoint problems with its own wording (and keeps the
|
||||
// `url::ParseError` as the io::Error source) while the shared constructor carries the
|
||||
// MinIO-derived texts. Unifying the two is a separate change, so the endpoint is
|
||||
// pre-validated here, after the credential and bucket checks so the order in which the
|
||||
// shared constructor would report the same failures is preserved.
|
||||
if conf.access_key.is_empty() || conf.secret_key.is_empty() {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if conf.bucket == "" {
|
||||
if conf.bucket.is_empty() {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
@@ -55,36 +76,29 @@ impl WarmBackendRustFS {
|
||||
Err(e) => return Err(std::io::Error::other(e)),
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: conf.access_key.clone(),
|
||||
secret_access_key: conf.secret_key.clone(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: conf.region.clone(),
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("endpoint URL must include a host"))?;
|
||||
let client = TransitionClient::new(&format!("{host}:{}", u.port().unwrap_or(default_port)), opts, "rustfs").await?;
|
||||
if u.host_str().is_none() {
|
||||
return Err(std::io::Error::other("endpoint URL must include a host"));
|
||||
}
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(Self(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
}))
|
||||
Ok(Self(
|
||||
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
|
||||
endpoint: &conf.endpoint,
|
||||
access_key: &conf.access_key,
|
||||
secret_key: &conf.secret_key,
|
||||
bucket: &conf.bucket,
|
||||
prefix: &conf.prefix,
|
||||
region: &conf.region,
|
||||
// RustFS tier endpoints are path-style, so bucket addressing stays on
|
||||
// `BucketLookupAuto`; pinning DNS here would break those endpoints.
|
||||
bucket_lookup: BucketLookupType::BucketLookupAuto,
|
||||
provider_tag: "rustfs",
|
||||
// Debug-only, env-gated loopback exception for this provider's own e2e tier
|
||||
// tests (rustfs/rustfs#6773); every other provider passes plain
|
||||
// `validate_outbound_url`.
|
||||
validate_endpoint: validate_rustfs_tier_endpoint,
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -97,7 +111,7 @@ impl WarmBackend for WarmBackendRustFS {
|
||||
length: i64,
|
||||
meta: HashMap<String, String>,
|
||||
) -> Result<String, std::io::Error> {
|
||||
let part_size = optimal_part_size(length)?;
|
||||
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
|
||||
let client = self.0.client.clone();
|
||||
let res = client
|
||||
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
|
||||
@@ -146,27 +160,6 @@ impl crate::services::tier::warm_backend::TransitionCandidateReconciler for Warm
|
||||
}
|
||||
}
|
||||
|
||||
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
|
||||
let configured_part_size = MIN_PART_SIZE;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(MIN_PART_SIZE);
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use futures::FutureExt;
|
||||
@@ -197,4 +190,23 @@ mod tests {
|
||||
};
|
||||
assert!(err.to_string().contains("host"), "expected host validation error, got: {err}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_network_setup() {
|
||||
let conf = rustfs_tier("https://127.0.0.1:9000");
|
||||
|
||||
match WarmBackendRustFS::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed")),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn loopback_opt_in_does_not_allow_other_private_endpoints() {
|
||||
let loopback = url::Url::parse("https://127.0.0.1:9000").unwrap();
|
||||
assert!(validate_rustfs_tier_endpoint_inner(&loopback, true).is_ok());
|
||||
|
||||
let private = url::Url::parse("https://10.0.0.1:9000").unwrap();
|
||||
assert!(validate_rustfs_tier_endpoint_inner(&private, true).is_err());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -19,76 +19,38 @@
|
||||
#![allow(clippy::all)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::services::tier::{
|
||||
tier_config::TierTencent,
|
||||
warm_backend::{WarmBackend, WarmBackendGetOpts, build_transition_put_options},
|
||||
warm_backend::{
|
||||
S3CompatibleWarmBackendParams, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
|
||||
new_s3_compatible_warm_backend, optimal_part_size,
|
||||
},
|
||||
warm_backend_s3::WarmBackendS3,
|
||||
};
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::PutObjectOptions,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
transition_api::{BucketLookupType, Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
|
||||
};
|
||||
use tracing::warn;
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
|
||||
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
const MAX_PARTS_COUNT: i64 = 10000;
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
pub struct WarmBackendTencent(WarmBackendS3);
|
||||
|
||||
impl WarmBackendTencent {
|
||||
pub async fn new(conf: &TierTencent, tier: &str) -> Result<Self, std::io::Error> {
|
||||
if conf.access_key == "" || conf.secret_key == "" {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if conf.bucket == "" {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
let u = match url::Url::parse(&conf.endpoint) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
return Err(std::io::Error::other(e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: conf.access_key.clone(),
|
||||
secret_access_key: conf.secret_key.clone(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: conf.region.clone(),
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
|
||||
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "tencent").await?;
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(Self(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
}))
|
||||
Ok(Self(
|
||||
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
|
||||
endpoint: &conf.endpoint,
|
||||
access_key: &conf.access_key,
|
||||
secret_key: &conf.secret_key,
|
||||
bucket: &conf.bucket,
|
||||
prefix: &conf.prefix,
|
||||
region: &conf.region,
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
provider_tag: "tencent",
|
||||
validate_endpoint: validate_outbound_url,
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -101,7 +63,7 @@ impl WarmBackend for WarmBackendTencent {
|
||||
length: i64,
|
||||
meta: HashMap<String, String>,
|
||||
) -> Result<String, std::io::Error> {
|
||||
let part_size = optimal_part_size(length)?;
|
||||
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
|
||||
let client = self.0.client.clone();
|
||||
let res = client
|
||||
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
|
||||
@@ -132,23 +94,29 @@ impl WarmBackend for WarmBackendTencent {
|
||||
}
|
||||
}
|
||||
|
||||
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::services::tier::tier_config::TierTencent;
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
/// The SSRF guard itself is exercised once, generically, in
|
||||
/// `warm_backend::tests` (see backlog#2040/backlog#2041 and
|
||||
/// rustfs/rustfs#6764) — this test only pins that this provider's
|
||||
/// production constructor really is wired through that shared path.
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_network_setup() {
|
||||
let conf = TierTencent {
|
||||
endpoint: "https://127.0.0.1:9000".to_string(),
|
||||
bucket: "tier-bucket".to_string(),
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
region: "us-east-1".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let configured_part_size = MIN_PART_SIZE;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(MIN_PART_SIZE);
|
||||
match WarmBackendTencent::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed")),
|
||||
}
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
@@ -5696,6 +5696,8 @@ impl SetDisks {
|
||||
{
|
||||
let grace = dangling_delete_grace();
|
||||
if !grace.is_zero() && OffsetDateTime::now_utc() - mod_time < grace {
|
||||
let elapsed = OffsetDateTime::now_utc() - mod_time;
|
||||
let retry_after_secs = grace.saturating_sub(elapsed).whole_seconds().max(0);
|
||||
info!(
|
||||
bucket = bucket,
|
||||
object = object,
|
||||
@@ -5703,7 +5705,7 @@ impl SetDisks {
|
||||
grace_secs = grace.whole_seconds(),
|
||||
"skipping dangling-object deletion within grace window"
|
||||
);
|
||||
return Err(DiskError::ErasureReadQuorum);
|
||||
return Err(DiskError::dangling_delete_grace(retry_after_secs, grace.whole_seconds()));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6799,6 +6801,7 @@ pub(in crate::set_disk) mod rename_fanout_barrier {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use crate::disk::error::HEAL_DANGLING_DELETE_GRACE_MESSAGE;
|
||||
use crate::disk::local::{DurabilityMode, durability_mode_override};
|
||||
|
||||
use super::*;
|
||||
@@ -10746,7 +10749,13 @@ mod tests {
|
||||
let object = "object";
|
||||
let (_dir, disk) = read_multiple_test_disk(bucket, &[]).await;
|
||||
let set = io_primitives_test_set(vec![Some(disk.clone()), None, None], 1).await;
|
||||
let mut fi = metadata_test_fileinfo(object);
|
||||
let mut fi = FileInfo::new(object, 2, 1);
|
||||
fi.volume = bucket.to_string();
|
||||
fi.name = object.to_string();
|
||||
fi.size = 1;
|
||||
fi.erasure.index = 1;
|
||||
fi.metadata.insert("etag".to_string(), "etag-1".to_string());
|
||||
fi.add_object_part(1, "part-etag-1".to_string(), 1, None, 1, None, None);
|
||||
fi.mod_time = Some(OffsetDateTime::now_utc());
|
||||
disk.write_metadata(bucket, bucket, object, fi.clone())
|
||||
.await
|
||||
@@ -10764,7 +10773,15 @@ mod tests {
|
||||
.await
|
||||
.expect_err("recent dangling metadata must stay protected by grace");
|
||||
|
||||
assert_eq!(err, DiskError::ErasureReadQuorum);
|
||||
let message = err.to_string();
|
||||
assert!(
|
||||
message.contains(HEAL_DANGLING_DELETE_GRACE_MESSAGE),
|
||||
"grace-protected dangling cleanup must explain the deferred delete: {message}"
|
||||
);
|
||||
assert!(
|
||||
message.contains("retry_after_secs="),
|
||||
"grace-protected dangling cleanup must include retry timing: {message}"
|
||||
);
|
||||
disk.read_all(bucket, &path_join_buf(&[object, STORAGE_FORMAT_FILE]))
|
||||
.await
|
||||
.expect("metadata should remain during dangling grace");
|
||||
|
||||
@@ -575,18 +575,12 @@ impl SetDisks {
|
||||
meta.metadata.keys().any(|name| http::is_object_encryption_marker(name))
|
||||
}
|
||||
|
||||
fn starts_with_ignore_ascii_case(value: &str, prefix: &str) -> bool {
|
||||
value
|
||||
.get(..prefix.len())
|
||||
.is_some_and(|value_prefix| value_prefix.eq_ignore_ascii_case(prefix))
|
||||
}
|
||||
|
||||
fn internal_metadata_suffix(name: &str) -> Option<&str> {
|
||||
name.get(http::RUSTFS_INTERNAL_PREFIX.len()..)
|
||||
.filter(|_| Self::starts_with_ignore_ascii_case(name, http::RUSTFS_INTERNAL_PREFIX))
|
||||
.filter(|_| http::starts_with_ignore_ascii_case(name, http::RUSTFS_INTERNAL_PREFIX))
|
||||
.or_else(|| {
|
||||
name.get(http::MINIO_INTERNAL_PREFIX.len()..)
|
||||
.filter(|_| Self::starts_with_ignore_ascii_case(name, http::MINIO_INTERNAL_PREFIX))
|
||||
.filter(|_| http::starts_with_ignore_ascii_case(name, http::MINIO_INTERNAL_PREFIX))
|
||||
})
|
||||
}
|
||||
|
||||
@@ -604,9 +598,9 @@ impl SetDisks {
|
||||
|| suffix.eq_ignore_ascii_case(http::SUFFIX_REPLICATION_STATUS)
|
||||
|| suffix.eq_ignore_ascii_case(http::SUFFIX_REPLICATION_TIMESTAMP)
|
||||
|| suffix.eq_ignore_ascii_case(http::SUFFIX_PURGESTATUS)
|
||||
|| Self::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_RESET_ARN_PREFIX)
|
||||
|| http::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_RESET_ARN_PREFIX)
|
||||
// Raw compatibility keys are normalized and hashed separately below.
|
||||
|| Self::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX)
|
||||
|| http::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX)
|
||||
}
|
||||
|
||||
fn update_hash_quorum_metadata_map(hasher: &mut Sha256, entries: &HashMap<String, String>) {
|
||||
@@ -1590,6 +1584,41 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
/// Guards the switch to `rustfs_utils::http::starts_with_ignore_ascii_case`:
|
||||
/// internal prefixes must keep matching case-insensitively, and keys shorter
|
||||
/// than the prefix must keep being rejected. Misclassifying either way leaks
|
||||
/// internal metadata into the quorum hash (or drops it out of it).
|
||||
#[test]
|
||||
fn internal_metadata_suffix_is_prefix_case_insensitive_and_rejects_short_keys() {
|
||||
assert_eq!(
|
||||
SetDisks::internal_metadata_suffix("X-RustFS-Internal-Replica-Status"),
|
||||
Some("Replica-Status"),
|
||||
"mixed-case RustFS prefix must match and preserve the suffix casing"
|
||||
);
|
||||
assert_eq!(
|
||||
SetDisks::internal_metadata_suffix("X-MINIO-INTERNAL-replica-status"),
|
||||
Some("replica-status"),
|
||||
"mixed-case MinIO prefix must match"
|
||||
);
|
||||
assert_eq!(SetDisks::internal_metadata_suffix(http::RUSTFS_INTERNAL_PREFIX), Some(""));
|
||||
|
||||
// Keys shorter than either prefix, and non-internal keys, stay unmatched.
|
||||
assert_eq!(SetDisks::internal_metadata_suffix(""), None);
|
||||
assert_eq!(SetDisks::internal_metadata_suffix("x-rustfs-interna"), None);
|
||||
assert_eq!(SetDisks::internal_metadata_suffix("x-minio-interna"), None);
|
||||
assert_eq!(SetDisks::internal_metadata_suffix("x-amz-meta-custom"), None);
|
||||
|
||||
// The suffix-prefix comparisons behind the classifier follow the same rules.
|
||||
assert!(SetDisks::is_replication_quorum_metadata_key(
|
||||
"X-RustFS-Internal-Replication-Reset-arn:rustfs:replication::target:bucket"
|
||||
));
|
||||
assert!(SetDisks::is_replication_quorum_metadata_key(
|
||||
"X-Minio-Internal-Replication-Delete-Marker-Version-arn:rustfs:replication::target:bucket"
|
||||
));
|
||||
assert!(!SetDisks::is_replication_quorum_metadata_key("x-rustfs-interna"));
|
||||
assert!(!SetDisks::is_replication_quorum_metadata_key("x-rustfs-internal-replication-res"));
|
||||
}
|
||||
|
||||
/// rustfs#5801: parity counts outside [0, total_shards] come from corrupt
|
||||
/// or foreign metadata and must be treated as invalid entries instead of
|
||||
/// clamped values that poison `common_parity`'s occurrence counting.
|
||||
|
||||
@@ -740,6 +740,10 @@ const ENV_RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY: &str = "RUSTFS_GET_SMALL_OBJECT
|
||||
const DEFAULT_RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY: bool = true;
|
||||
const ENV_RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY_THRESHOLD: &str = "RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY_THRESHOLD";
|
||||
const DEFAULT_RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY_THRESHOLD: usize = 128 * 1024;
|
||||
// Bound the in-memory inline decoder independently from the configurable
|
||||
// storage-class admission policy. This protects its Vec allocation while
|
||||
// allowing explicitly configured EC layouts to inline objects above 128 KiB.
|
||||
const INLINE_FAST_PATH_MAX_OBJECT_SIZE: i64 = 1024 * 1024;
|
||||
|
||||
// --- Metadata Early-Stop Configuration ---
|
||||
|
||||
@@ -2285,6 +2289,14 @@ fn get_codec_streaming_config() -> GetCodecStreamingConfig {
|
||||
}
|
||||
}
|
||||
|
||||
/// Shared kill-switch and compatibility guard for all non-duplex GET readers.
|
||||
/// Size-specific readers may have independent rollout gates, but they must not
|
||||
/// bypass these emergency controls.
|
||||
pub(super) fn is_get_codec_streaming_base_enabled() -> bool {
|
||||
let config = get_codec_streaming_config();
|
||||
config.enabled && config.body_compat_confirmed && config.header_compat_confirmed
|
||||
}
|
||||
|
||||
fn get_codec_streaming_engine() -> GetCodecStreamingEngine {
|
||||
get_codec_streaming_config().engine
|
||||
}
|
||||
@@ -2345,6 +2357,7 @@ enum GetCodecStreamingFallbackReason {
|
||||
ReadQuorumNotSafe,
|
||||
MultipartPartLimit,
|
||||
CopySourceDemandBound,
|
||||
InvalidMetadataShape,
|
||||
}
|
||||
|
||||
impl GetCodecStreamingFallbackReason {
|
||||
@@ -2367,6 +2380,7 @@ impl GetCodecStreamingFallbackReason {
|
||||
Self::ReadQuorumNotSafe => "read_quorum_not_safe",
|
||||
Self::MultipartPartLimit => "multipart_part_limit",
|
||||
Self::CopySourceDemandBound => "copy_source_demand_bound",
|
||||
Self::InvalidMetadataShape => "invalid_metadata_shape",
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2426,6 +2440,7 @@ enum GetDirectMemoryFallbackReason {
|
||||
Remote,
|
||||
ObjectInfoMultipart,
|
||||
FileInfoMultipart,
|
||||
MetadataShape,
|
||||
InvalidSize,
|
||||
SizeMismatch,
|
||||
AboveThreshold,
|
||||
@@ -2451,6 +2466,7 @@ impl GetDirectMemoryFallbackReason {
|
||||
Self::Remote => "remote",
|
||||
Self::ObjectInfoMultipart => "object_info_multipart",
|
||||
Self::FileInfoMultipart => "file_info_multipart",
|
||||
Self::MetadataShape => "metadata_shape",
|
||||
Self::InvalidSize => "invalid_size",
|
||||
Self::SizeMismatch => "size_mismatch",
|
||||
Self::AboveThreshold => "above_threshold",
|
||||
@@ -2499,10 +2515,42 @@ fn record_get_object_reader_path_observation(
|
||||
object_class: GetCodecStreamingObjectClass,
|
||||
size_bucket: &'static str,
|
||||
) {
|
||||
#[cfg(test)]
|
||||
LAST_GET_OBJECT_READER_PATH.store(
|
||||
match path {
|
||||
crate::set_disk::read::GET_OBJECT_PATH_MID_SIZE_STREAMING => 1,
|
||||
GET_OBJECT_PATH_DIRECT_MEMORY => 2,
|
||||
GET_OBJECT_PATH_INLINE_DIRECT => 3,
|
||||
GET_OBJECT_PATH_BODY_CACHE => 4,
|
||||
GET_OBJECT_PATH_CODEC_STREAMING => 5,
|
||||
GET_OBJECT_PATH_REMOTE_TRANSITION => 6,
|
||||
GET_OBJECT_PATH_EMPTY => 7,
|
||||
_ => 255,
|
||||
},
|
||||
Ordering::Relaxed,
|
||||
);
|
||||
rustfs_io_metrics::record_get_object_reader_path(path);
|
||||
rustfs_io_metrics::record_get_object_reader_path_by_size(path, object_class.as_str(), size_bucket);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
static LAST_GET_OBJECT_READER_PATH: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn reset_test_get_object_reader_path() {
|
||||
LAST_GET_OBJECT_READER_PATH.store(0, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn test_get_object_reader_selected_mid_size() -> bool {
|
||||
LAST_GET_OBJECT_READER_PATH.load(Ordering::Relaxed) == 1
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn test_get_object_reader_path_id() -> u64 {
|
||||
LAST_GET_OBJECT_READER_PATH.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
fn classify_get_codec_streaming_object_class(
|
||||
range: &Option<HTTPRangeSpec>,
|
||||
object_info: &ObjectInfo,
|
||||
@@ -2547,6 +2595,26 @@ fn get_small_object_direct_memory_decision_with_threshold(
|
||||
opts: &ObjectOptions,
|
||||
enabled: bool,
|
||||
threshold: usize,
|
||||
) -> GetDirectMemoryDecision {
|
||||
get_small_object_direct_memory_decision_with_threshold_and_plan(
|
||||
range,
|
||||
object_info,
|
||||
fi,
|
||||
opts,
|
||||
enabled,
|
||||
threshold,
|
||||
ReadPathPlan::new(object_info, fi),
|
||||
)
|
||||
}
|
||||
|
||||
fn get_small_object_direct_memory_decision_with_threshold_and_plan(
|
||||
range: &Option<HTTPRangeSpec>,
|
||||
object_info: &ObjectInfo,
|
||||
fi: &FileInfo,
|
||||
opts: &ObjectOptions,
|
||||
enabled: bool,
|
||||
threshold: usize,
|
||||
plan: ReadPathPlan,
|
||||
) -> GetDirectMemoryDecision {
|
||||
if !enabled {
|
||||
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::Disabled);
|
||||
@@ -2605,14 +2673,24 @@ fn get_small_object_direct_memory_decision_with_threshold(
|
||||
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::FileInfoMultipart);
|
||||
}
|
||||
|
||||
let Ok(object_size) = usize::try_from(fi.size) else {
|
||||
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::InvalidSize);
|
||||
if object_info.size != fi.size {
|
||||
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::SizeMismatch);
|
||||
}
|
||||
let Some(shape) = plan.shape() else {
|
||||
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::MetadataShape);
|
||||
};
|
||||
let object_size = shape.object_size;
|
||||
if object_size == 0 {
|
||||
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::InvalidSize);
|
||||
}
|
||||
if object_info.size != fi.size {
|
||||
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::SizeMismatch);
|
||||
if !plan.is_plain() {
|
||||
if object_info.is_encrypted() || fi.metadata.keys().any(|key| is_object_encryption_marker(key)) {
|
||||
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::Encrypted);
|
||||
}
|
||||
if object_info.is_compressed() || fi.is_compressed() {
|
||||
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::Compressed);
|
||||
}
|
||||
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::Remote);
|
||||
}
|
||||
if object_size > threshold {
|
||||
return GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::AboveThreshold);
|
||||
@@ -2621,6 +2699,7 @@ fn get_small_object_direct_memory_decision_with_threshold(
|
||||
GetDirectMemoryDecision::Use { object_size }
|
||||
}
|
||||
|
||||
#[allow(dead_code, reason = "asserted by this file's gate tests")]
|
||||
fn get_small_object_direct_memory_decision(
|
||||
range: &Option<HTTPRangeSpec>,
|
||||
object_info: &ObjectInfo,
|
||||
@@ -2655,6 +2734,7 @@ fn should_prefer_codec_streaming_data_blocks_first_reader_setup(
|
||||
max_size > 0 && object_size <= max_size
|
||||
}
|
||||
|
||||
#[allow(dead_code, reason = "asserted by this file's gate tests")]
|
||||
fn get_codec_streaming_reader_gate(
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
@@ -2663,6 +2743,29 @@ fn get_codec_streaming_reader_gate(
|
||||
object_info: &ObjectInfo,
|
||||
fi: &FileInfo,
|
||||
lock_optimization_enabled: bool,
|
||||
) -> GetCodecStreamingGate {
|
||||
get_codec_streaming_reader_gate_with_plan(
|
||||
bucket,
|
||||
object,
|
||||
part_number,
|
||||
object_class,
|
||||
object_info,
|
||||
fi,
|
||||
lock_optimization_enabled,
|
||||
ReadPathPlan::new(object_info, fi),
|
||||
)
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments, reason = "keeps the hot-path gate inputs explicit")]
|
||||
fn get_codec_streaming_reader_gate_with_plan(
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
part_number: Option<usize>,
|
||||
object_class: GetCodecStreamingObjectClass,
|
||||
object_info: &ObjectInfo,
|
||||
fi: &FileInfo,
|
||||
lock_optimization_enabled: bool,
|
||||
plan: ReadPathPlan,
|
||||
) -> GetCodecStreamingGate {
|
||||
let config = get_codec_streaming_config();
|
||||
|
||||
@@ -2778,6 +2881,22 @@ fn get_codec_streaming_reader_gate(
|
||||
};
|
||||
}
|
||||
}
|
||||
if object_class == GetCodecStreamingObjectClass::PlainSinglePart {
|
||||
let Some(_shape) = plan.shape() else {
|
||||
return GetCodecStreamingGate {
|
||||
object_class,
|
||||
decision: GetCodecStreamingDecision::Fallback(GetCodecStreamingFallbackReason::InvalidMetadataShape),
|
||||
prefer_data_blocks_first_reader_setup: false,
|
||||
};
|
||||
};
|
||||
if !plan.is_plain() {
|
||||
return GetCodecStreamingGate {
|
||||
object_class,
|
||||
decision: GetCodecStreamingDecision::Fallback(GetCodecStreamingFallbackReason::InvalidMetadataShape),
|
||||
prefer_data_blocks_first_reader_setup: false,
|
||||
};
|
||||
}
|
||||
}
|
||||
let Ok(min_size) = i64::try_from(config.min_size) else {
|
||||
return GetCodecStreamingGate {
|
||||
object_class,
|
||||
@@ -4062,6 +4181,80 @@ fn object_fits_single_block(object_size: i64, block_size: usize) -> bool {
|
||||
}
|
||||
}
|
||||
|
||||
/// The common metadata contract consumed by all bounded GET fast paths.
|
||||
///
|
||||
/// `ObjectInfo` is assembled from `FileInfo`, but callers may also provide a
|
||||
/// prepared snapshot or metadata from an older peer. Never let either copy
|
||||
/// independently decide that a request is safe: a disagreement must fall
|
||||
/// back to the regular reader. The returned size is the only size used for
|
||||
/// fast-path allocation and reader setup.
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub(super) struct ReadPathShape {
|
||||
pub(super) object_size: usize,
|
||||
}
|
||||
|
||||
impl ReadPathShape {
|
||||
pub(super) fn is_plain(self, object_info: &ObjectInfo, fi: &FileInfo) -> bool {
|
||||
!object_info.is_encrypted()
|
||||
&& !fi.metadata.keys().any(|key| is_object_encryption_marker(key))
|
||||
&& !object_info.is_compressed()
|
||||
&& !fi.is_compressed()
|
||||
&& !object_info.is_remote()
|
||||
&& !fi.is_remote()
|
||||
}
|
||||
}
|
||||
|
||||
/// Request-local metadata decision reused by all small-object GET gates.
|
||||
///
|
||||
/// The metadata pair is immutable for the lifetime of `get_object_reader`, so
|
||||
/// validating it once avoids repeating part-array and transform scans on every
|
||||
/// fast-path predicate while keeping the trust boundary fail-closed.
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
struct ReadPathPlan {
|
||||
shape: Option<ReadPathShape>,
|
||||
plain: bool,
|
||||
}
|
||||
|
||||
impl ReadPathPlan {
|
||||
fn new(object_info: &ObjectInfo, fi: &FileInfo) -> Self {
|
||||
let shape = read_path_shape(object_info, fi);
|
||||
let plain = shape.is_some_and(|shape| shape.is_plain(object_info, fi));
|
||||
Self { shape, plain }
|
||||
}
|
||||
|
||||
const fn shape(self) -> Option<ReadPathShape> {
|
||||
self.shape
|
||||
}
|
||||
|
||||
const fn is_plain(self) -> bool {
|
||||
self.plain
|
||||
}
|
||||
}
|
||||
|
||||
/// Validate the single-part geometry shared by inline, direct-memory, and
|
||||
/// bounded mid-size readers. This is deliberately fail-closed: a stale or
|
||||
/// mixed-version `ObjectInfo`/`FileInfo` pair must use the legacy path rather
|
||||
/// than risk allocating or decoding with a mismatched size.
|
||||
pub(super) fn read_path_shape(object_info: &ObjectInfo, fi: &FileInfo) -> Option<ReadPathShape> {
|
||||
if object_info.parts.len() != 1 || fi.parts.len() != 1 || object_info.size < 0 || fi.size < 0 || object_info.size != fi.size {
|
||||
return None;
|
||||
}
|
||||
|
||||
let object_part = object_info.parts.first()?;
|
||||
let file_part = fi.parts.first()?;
|
||||
let object_size = usize::try_from(fi.size).ok()?;
|
||||
if object_part.number != file_part.number
|
||||
|| object_part.size != file_part.size
|
||||
|| object_part.actual_size != file_part.actual_size
|
||||
|| file_part.size != object_size
|
||||
|| file_part.actual_size != fi.size
|
||||
{
|
||||
return None;
|
||||
}
|
||||
|
||||
Some(ReadPathShape { object_size })
|
||||
}
|
||||
|
||||
fn should_use_inline_small_fast_path(is_inline_buffer: bool, object_size: i64, block_size: usize) -> bool {
|
||||
is_inline_buffer && object_fits_single_block(object_size, block_size)
|
||||
}
|
||||
@@ -4070,13 +4263,43 @@ fn should_use_single_block_non_inline_fast_path(is_inline_buffer: bool, object_s
|
||||
!is_inline_buffer && object_fits_single_block(object_size, block_size)
|
||||
}
|
||||
|
||||
#[allow(dead_code, reason = "asserted by this file's gate tests")]
|
||||
fn should_use_inline_fast_path(
|
||||
range: &Option<HTTPRangeSpec>,
|
||||
object_info: &ObjectInfo,
|
||||
fi: &FileInfo,
|
||||
opts: &ObjectOptions,
|
||||
) -> bool {
|
||||
object_info.is_inline_fast_path_eligible() && fi.data.is_some() && range.is_none() && opts.part_number.is_none()
|
||||
should_use_inline_fast_path_with_plan(range, object_info, fi, opts, ReadPathPlan::new(object_info, fi))
|
||||
}
|
||||
|
||||
fn should_use_inline_fast_path_with_plan(
|
||||
range: &Option<HTTPRangeSpec>,
|
||||
object_info: &ObjectInfo,
|
||||
fi: &FileInfo,
|
||||
opts: &ObjectOptions,
|
||||
plan: ReadPathPlan,
|
||||
) -> bool {
|
||||
if !object_info.is_inline_fast_path_eligible() || fi.data.is_none() || range.is_some() || opts.part_number.is_some() {
|
||||
return false;
|
||||
}
|
||||
|
||||
let Some(shape) = plan.shape() else {
|
||||
return false;
|
||||
};
|
||||
// The persisted marker is authoritative for the storage decision, but it
|
||||
// is still untrusted metadata at this boundary. Revalidate its geometry so
|
||||
// a stale/corrupt marker cannot route an oversized payload into the
|
||||
// in-memory decoder. The independent direct-memory reader remains capped
|
||||
// at 128 KiB below.
|
||||
if !plan.is_plain()
|
||||
|| shape.object_size > usize::try_from(INLINE_FAST_PATH_MAX_OBJECT_SIZE).expect("inline fast path limit fits usize")
|
||||
|| fi.erasure.data_blocks == 0
|
||||
|| fi.erasure.block_size == 0
|
||||
{
|
||||
return false;
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
enum SmallWritePath {
|
||||
@@ -4817,6 +5040,28 @@ fn should_force_delete_marker_for_missing_version(opts: &ObjectOptions) -> bool
|
||||
opts.delete_marker || ((opts.versioned || opts.version_suspended) && opts.version_id.is_none() && !opts.data_movement)
|
||||
}
|
||||
|
||||
/// Whether a plain client delete addressed to an explicit version removed an
|
||||
/// existing delete marker, so the response must carry delete-marker semantics
|
||||
/// (`x-amz-delete-marker: true` / `DeleteMarker` in `DeleteObjects` entries).
|
||||
///
|
||||
/// This is a response-side classification only. It must never feed the
|
||||
/// storage write shape: a delete request marked `deleted` with the null
|
||||
/// identity makes `FileMeta::delete_version` re-create the marker it just
|
||||
/// removed (that is the suspended-bucket "delete mints a marker" write
|
||||
/// path), which is why `resolve_delete_version_state` cannot simply report
|
||||
/// `delete_marker` for suspended buckets (issue #6745).
|
||||
///
|
||||
/// Purge/replica replication shapes are excluded, mirroring the clauses in
|
||||
/// `resolve_delete_version_state` that clear `delete_marker` for them.
|
||||
fn explicit_delete_removed_marker(opts: &ObjectOptions, goi: &ObjectInfo, version_found: bool) -> bool {
|
||||
opts.version_id.is_some()
|
||||
&& version_found
|
||||
&& goi.delete_marker
|
||||
&& goi.version_purge_status.is_empty()
|
||||
&& opts.version_purge_status().is_empty()
|
||||
&& opts.delete_marker_replication_status().is_empty()
|
||||
}
|
||||
|
||||
fn resolve_delete_version_state(opts: &ObjectOptions, goi: &ObjectInfo, version_found: bool) -> (bool, bool) {
|
||||
let mut mark_delete = goi.version_id.is_some() || ((opts.versioned || opts.version_suspended) && opts.version_id.is_none());
|
||||
let mut delete_marker = opts.versioned;
|
||||
@@ -7038,6 +7283,85 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_delete_version_state_keeps_null_marker_removal_write_shape_undeleted() {
|
||||
// Removing a null delete marker by explicit version id on a
|
||||
// versioning-suspended bucket must NOT mark the write request
|
||||
// `deleted`: `FileMeta::delete_version` re-creates a marker for
|
||||
// `deleted` requests carrying the null identity (the suspended-bucket
|
||||
// "delete mints a marker" path), which would resurrect the marker
|
||||
// being removed. The response-side marker semantics come from
|
||||
// `explicit_delete_removed_marker` instead (issue #6745).
|
||||
let opts = ObjectOptions {
|
||||
version_suspended: true,
|
||||
version_id: Some(Uuid::nil().to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let current = ObjectInfo {
|
||||
version_id: Some(Uuid::nil()),
|
||||
delete_marker: true,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let (mark_delete, delete_marker) = resolve_delete_version_state(&opts, ¤t, true);
|
||||
|
||||
assert!(!mark_delete);
|
||||
assert!(!delete_marker, "the storage write for a null-marker removal must stay undeleted");
|
||||
assert!(
|
||||
explicit_delete_removed_marker(&opts, ¤t, true),
|
||||
"the response must still report delete-marker semantics"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn explicit_delete_removed_marker_is_limited_to_plain_marker_removals() {
|
||||
let opts = ObjectOptions {
|
||||
versioned: true,
|
||||
version_id: Some(Uuid::new_v4().to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let marker = ObjectInfo {
|
||||
version_id: Some(Uuid::new_v4()),
|
||||
delete_marker: true,
|
||||
..Default::default()
|
||||
};
|
||||
assert!(explicit_delete_removed_marker(&opts, &marker, true));
|
||||
assert!(
|
||||
!explicit_delete_removed_marker(&opts, &marker, false),
|
||||
"missing versions are not marker removals"
|
||||
);
|
||||
|
||||
let data_version = ObjectInfo {
|
||||
version_id: marker.version_id,
|
||||
delete_marker: false,
|
||||
..Default::default()
|
||||
};
|
||||
assert!(!explicit_delete_removed_marker(&opts, &data_version, true));
|
||||
|
||||
let versionless = ObjectOptions {
|
||||
versioned: true,
|
||||
..Default::default()
|
||||
};
|
||||
assert!(
|
||||
!explicit_delete_removed_marker(&versionless, &marker, true),
|
||||
"marker creation (no version in the request) is not a removal"
|
||||
);
|
||||
|
||||
let replica_purge = ObjectOptions {
|
||||
versioned: true,
|
||||
version_id: Some(Uuid::new_v4().to_string()),
|
||||
delete_replication: Some(ReplicationState {
|
||||
replica_status: ReplicationStatusType::Replica,
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
assert!(
|
||||
!explicit_delete_removed_marker(&replica_purge, &marker, true),
|
||||
"replication purge shapes keep their existing response semantics"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_delete_version_state_keeps_delete_marker_for_replica_marker_creation() {
|
||||
let opts = ObjectOptions {
|
||||
@@ -10881,6 +11205,41 @@ mod tests {
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn read_path_shape_rejects_mismatched_metadata_copies_and_transforms() {
|
||||
let (object_info, fi, _) = direct_memory_test_metadata(1024);
|
||||
let shape = read_path_shape(&object_info, &fi).expect("matching metadata should have a valid shape");
|
||||
assert_eq!(shape.object_size, 1024);
|
||||
assert!(shape.is_plain(&object_info, &fi));
|
||||
|
||||
let mut bad_part_number = object_info.clone();
|
||||
Arc::make_mut(&mut bad_part_number.parts)[0].number = 2;
|
||||
assert!(read_path_shape(&bad_part_number, &fi).is_none());
|
||||
|
||||
let mut bad_part_size = fi.clone();
|
||||
bad_part_size.parts[0].size += 1;
|
||||
assert!(read_path_shape(&object_info, &bad_part_size).is_none());
|
||||
|
||||
let mut bad_actual_size = object_info.clone();
|
||||
Arc::make_mut(&mut bad_actual_size.parts)[0].actual_size += 1;
|
||||
assert!(read_path_shape(&bad_actual_size, &fi).is_none());
|
||||
|
||||
let mut compressed = fi.clone();
|
||||
insert_str(&mut compressed.metadata, SUFFIX_COMPRESSION, "zstd".to_string());
|
||||
let compressed_shape = read_path_shape(&object_info, &compressed).expect("geometry remains valid");
|
||||
assert!(!compressed_shape.is_plain(&object_info, &compressed));
|
||||
|
||||
let mut encrypted = fi;
|
||||
encrypted
|
||||
.metadata
|
||||
.insert("x-amz-server-side-encryption".to_string(), "AES256".to_string());
|
||||
assert!(
|
||||
!read_path_shape(&object_info, &encrypted)
|
||||
.expect("geometry remains valid")
|
||||
.is_plain(&object_info, &encrypted)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inline_fast_path_rejects_part_number_requests() {
|
||||
let (mut object_info, mut fi, opts) = direct_memory_test_metadata(1024);
|
||||
@@ -10894,6 +11253,68 @@ mod tests {
|
||||
assert!(!should_use_inline_fast_path(&None, &object_info, &fi, &part_opts));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inline_fast_path_allows_layout_specific_ec8_and_ec12_256kib_objects() {
|
||||
for (data_blocks, parity_blocks) in [(8, 4), (12, 4)] {
|
||||
let object_size = 256 * 1024_i64;
|
||||
let mut fi = FileInfo::new("bucket/object", data_blocks, parity_blocks);
|
||||
fi.size = object_size;
|
||||
fi.data = Some(Bytes::from_static(b"payload"));
|
||||
fi.add_object_part(1, String::new(), object_size as usize, None, object_size, None, None);
|
||||
let object_info = ObjectInfo {
|
||||
size: object_size,
|
||||
data_blocks,
|
||||
parity_blocks,
|
||||
inlined: true,
|
||||
parts: Arc::new(fi.parts.clone()),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
assert!(should_use_inline_fast_path(&None, &object_info, &fi, &ObjectOptions::default()));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inline_fast_path_rejects_oversized_or_corrupt_inline_markers() {
|
||||
for (data_blocks, parity_blocks) in [(1, 0), (8, 4), (12, 4)] {
|
||||
let object_size = 4 * 1024 * 1024_i64;
|
||||
let mut fi = FileInfo::new("bucket/object", data_blocks, parity_blocks);
|
||||
fi.size = object_size;
|
||||
fi.data = Some(Bytes::from_static(b"payload"));
|
||||
fi.add_object_part(1, String::new(), object_size as usize, None, object_size, None, None);
|
||||
let object_info = ObjectInfo {
|
||||
size: object_size,
|
||||
data_blocks,
|
||||
parity_blocks,
|
||||
inlined: true,
|
||||
parts: Arc::new(fi.parts.clone()),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
assert!(
|
||||
object_info.is_inline_fast_path_eligible(),
|
||||
"the marker and object shape alone must not be the final trust boundary"
|
||||
);
|
||||
assert!(!should_use_inline_fast_path(&None, &object_info, &fi, &ObjectOptions::default()));
|
||||
}
|
||||
|
||||
// A malformed erasure geometry must fail closed before shard-size
|
||||
// arithmetic, even when an inline marker and payload are present.
|
||||
let (mut object_info, mut fi, opts) = direct_memory_test_metadata(4 * 1024 * 1024);
|
||||
object_info.inlined = true;
|
||||
fi.data = Some(Bytes::from_static(b"payload"));
|
||||
fi.erasure.data_blocks = 0;
|
||||
assert!(!should_use_inline_fast_path(&None, &object_info, &fi, &opts));
|
||||
|
||||
// A mismatched plain part size must not allow the large object size to
|
||||
// reach the inline decoder's `Vec::with_capacity` allocation.
|
||||
let (mut object_info, mut fi, opts) = direct_memory_test_metadata(4 * 1024 * 1024);
|
||||
object_info.inlined = true;
|
||||
fi.data = Some(Bytes::from_static(b"payload"));
|
||||
fi.parts[0].actual_size = 0;
|
||||
assert!(!should_use_inline_fast_path(&None, &object_info, &fi, &opts));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn small_object_direct_memory_decision_reports_bounded_reasons() {
|
||||
let (object_info, fi, opts) = direct_memory_test_metadata(1024);
|
||||
@@ -10967,6 +11388,13 @@ mod tests {
|
||||
get_small_object_direct_memory_decision_with_threshold(&None, &object_info, &fi, &opts, true, 128 * 1024),
|
||||
GetDirectMemoryDecision::Use { object_size: 1024 }
|
||||
);
|
||||
|
||||
let mut corrupt_part = fi;
|
||||
corrupt_part.parts[0].actual_size += 1;
|
||||
assert_eq!(
|
||||
get_small_object_direct_memory_decision_with_threshold(&None, &object_info, &corrupt_part, &opts, true, 128 * 1024),
|
||||
GetDirectMemoryDecision::Fallback(GetDirectMemoryFallbackReason::MetadataShape)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -3543,7 +3543,20 @@ mod heal_result_report_tests {
|
||||
.await
|
||||
.expect("grace-protected dangling metadata should return a typed heal result");
|
||||
|
||||
assert_eq!(error, Some(DiskError::ErasureReadQuorum));
|
||||
let error = error.expect("grace-protected dangling metadata should be reported as deferred");
|
||||
assert!(
|
||||
error.is_dangling_delete_grace(),
|
||||
"grace-protected dangling metadata should keep a typed deferred-cleanup marker: {error}"
|
||||
);
|
||||
let message = error.to_string();
|
||||
assert!(
|
||||
message.contains("dangling object deletion deferred by heal grace window"),
|
||||
"grace-protected dangling metadata should explain that cleanup was deferred: {message}"
|
||||
);
|
||||
assert!(
|
||||
message.contains("retry_after_secs="),
|
||||
"grace-protected dangling metadata should include retry timing: {message}"
|
||||
);
|
||||
assert!(
|
||||
temp_dirs[0]
|
||||
.path()
|
||||
|
||||
@@ -84,11 +84,13 @@ use rustfs_rio::TryGetIndex;
|
||||
use rustfs_utils::http::SSEC_ALGORITHM_HEADER;
|
||||
#[cfg(test)]
|
||||
use rustfs_utils::http::SUFFIX_COMPRESSION;
|
||||
use rustfs_utils::http::{SUFFIX_MAX_TOTAL_OBJECT_SIZE, get_consistent_str};
|
||||
use std::future::Future;
|
||||
#[cfg(test)]
|
||||
use std::sync::atomic::AtomicBool;
|
||||
#[cfg(any(test, feature = "test-util"))]
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::sync::{Mutex, OnceLock};
|
||||
#[cfg(any(test, feature = "test-util"))]
|
||||
use std::time::Duration;
|
||||
#[cfg(test)]
|
||||
@@ -97,6 +99,83 @@ use tokio::task::JoinSet;
|
||||
|
||||
const MULTIPART_LIST_IO_CONCURRENCY: usize = 16;
|
||||
|
||||
static CAPPED_MULTIPART_STAGING: OnceLock<Mutex<HashMap<String, Arc<tokio::sync::Semaphore>>>> = OnceLock::new();
|
||||
|
||||
struct CappedMultipartStagingGuard {
|
||||
upload_id_path: String,
|
||||
permit: Option<tokio::sync::OwnedSemaphorePermit>,
|
||||
}
|
||||
|
||||
impl Drop for CappedMultipartStagingGuard {
|
||||
fn drop(&mut self) {
|
||||
// Release the permit before checking the Arc count so a concurrent
|
||||
// Abort/Complete cleanup can remove the now-unused map entry.
|
||||
self.permit.take();
|
||||
remove_capped_multipart_staging_semaphore(&self.upload_id_path);
|
||||
}
|
||||
}
|
||||
|
||||
fn capped_multipart_staging_semaphore(upload_id_path: &str) -> Arc<tokio::sync::Semaphore> {
|
||||
CAPPED_MULTIPART_STAGING
|
||||
.get_or_init(|| Mutex::new(HashMap::new()))
|
||||
.lock()
|
||||
.expect("capped multipart staging semaphore map should not be poisoned")
|
||||
.entry(upload_id_path.to_owned())
|
||||
.or_insert_with(|| Arc::new(tokio::sync::Semaphore::new(1)))
|
||||
.clone()
|
||||
}
|
||||
|
||||
fn remove_capped_multipart_staging_semaphore(upload_id_path: &str) {
|
||||
if let Some(map) = CAPPED_MULTIPART_STAGING.get() {
|
||||
let mut map = map
|
||||
.lock()
|
||||
.expect("capped multipart staging semaphore map should not be poisoned");
|
||||
let removable = map
|
||||
.get(upload_id_path)
|
||||
.is_some_and(|semaphore| Arc::strong_count(semaphore) == 1);
|
||||
if removable {
|
||||
map.remove(upload_id_path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn multipart_size_limit_from_metadata(metadata: &HashMap<String, String>) -> Result<Option<u64>> {
|
||||
if !contains_key_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE) {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
let Some(value) = get_consistent_str(metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE) else {
|
||||
return Err(Error::InvalidArgument(
|
||||
"multipart upload".to_string(),
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
|
||||
"missing or conflicting internal size limit".to_string(),
|
||||
));
|
||||
};
|
||||
|
||||
let limit = value.parse::<u64>().map_err(|_| {
|
||||
Error::InvalidArgument(
|
||||
"multipart upload".to_string(),
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
|
||||
"invalid internal size limit".to_string(),
|
||||
)
|
||||
})?;
|
||||
Ok(Some(limit))
|
||||
}
|
||||
|
||||
fn admitted_multipart_size(current: u64, candidate: u64, limit: u64) -> Result<u64> {
|
||||
let total = current.checked_add(candidate).ok_or_else(|| {
|
||||
Error::InvalidArgument(
|
||||
"multipart upload".to_string(),
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
|
||||
"logical size overflow".to_string(),
|
||||
)
|
||||
})?;
|
||||
if total > limit {
|
||||
return Err(Error::EntityTooLarge(total, limit));
|
||||
}
|
||||
Ok(total)
|
||||
}
|
||||
|
||||
pub(crate) struct StaleMultipartCleanupGuard {
|
||||
file_info: FileInfo,
|
||||
upload_path: String,
|
||||
@@ -115,8 +194,13 @@ impl StaleMultipartCleanupGuard {
|
||||
|
||||
pub(crate) async fn delete(self, set: &SetDisks) -> Result<()> {
|
||||
fence_commit_on_lock_loss(Some(&self.lock_guard), "stale_multipart_cleanup", &self.upload_path)?;
|
||||
set.delete_all_with_quorum(RUSTFS_META_MULTIPART_BUCKET, &self.upload_path, self.write_quorum)
|
||||
.await
|
||||
let result = set
|
||||
.delete_all_with_quorum(RUSTFS_META_MULTIPART_BUCKET, &self.upload_path, self.write_quorum)
|
||||
.await;
|
||||
if result.is_ok() {
|
||||
remove_capped_multipart_staging_semaphore(&self.upload_path);
|
||||
}
|
||||
result
|
||||
}
|
||||
}
|
||||
|
||||
@@ -635,6 +719,61 @@ async fn multipart_upload_paths_on_disk(disk: DiskStore, bucket: &str) -> disk::
|
||||
}
|
||||
|
||||
impl SetDisks {
|
||||
async fn current_multipart_logical_size(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
upload_id: &str,
|
||||
upload_id_path: &str,
|
||||
fi: &FileInfo,
|
||||
replacing_part: usize,
|
||||
) -> Result<u64> {
|
||||
let online_disks = self.get_disks_internal().await;
|
||||
let read_quorum = fi.read_quorum(self.default_read_quorum());
|
||||
let part_path = format!(
|
||||
"{}{}",
|
||||
path_join_buf(&[
|
||||
upload_id_path,
|
||||
fi.data_dir.map(|v| v.to_string()).unwrap_or_default().as_str(),
|
||||
]),
|
||||
SLASH_SEPARATOR
|
||||
);
|
||||
let part_numbers = match Self::list_parts(&online_disks, &part_path, read_quorum).await {
|
||||
Ok(parts) => parts,
|
||||
Err(DiskError::FileNotFound) => return Ok(0),
|
||||
Err(err) => return Err(to_object_err(err.into(), vec![bucket, object, upload_id])),
|
||||
};
|
||||
if part_numbers.is_empty() {
|
||||
return Ok(0);
|
||||
}
|
||||
let part_meta_paths = part_numbers
|
||||
.iter()
|
||||
.map(|number| format!("{part_path}part.{number}.meta"))
|
||||
.collect::<Vec<_>>();
|
||||
let existing_parts =
|
||||
Self::read_parts(&online_disks, RUSTFS_META_MULTIPART_BUCKET, &part_meta_paths, &part_numbers, read_quorum)
|
||||
.await
|
||||
.map_err(|err| to_object_err(err.into(), vec![bucket, object, upload_id]))?;
|
||||
|
||||
existing_parts.into_iter().try_fold(0_u64, |total, part| {
|
||||
if part.error.is_some() || part.number == replacing_part {
|
||||
return if part.error.is_some() {
|
||||
Err(Error::PartMissingOrCorrupt)
|
||||
} else {
|
||||
Ok(total)
|
||||
};
|
||||
}
|
||||
let part_size = u64::try_from(part.actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
|
||||
total.checked_add(part_size).ok_or_else(|| {
|
||||
Error::InvalidArgument(
|
||||
"multipart upload".to_string(),
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
|
||||
"logical size overflow".to_string(),
|
||||
)
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
async fn discover_multipart_upload_paths(
|
||||
&self,
|
||||
orig_bucket: &str,
|
||||
@@ -1130,9 +1269,18 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
crate::hp_guard!("SetDisks::put_object_part");
|
||||
let upload_id_path = Self::get_multipart_upload_dir(bucket, object, upload_id, opts.data_movement);
|
||||
|
||||
let (fi, _) = self
|
||||
let (fi, _) = match self
|
||||
.check_upload_id_exists_with_opts(bucket, object, upload_id, true, opts)
|
||||
.await?;
|
||||
.await
|
||||
{
|
||||
Ok(value) => value,
|
||||
Err(err @ Error::InvalidUploadID(..)) => {
|
||||
remove_capped_multipart_staging_semaphore(&upload_id_path);
|
||||
return Err(err);
|
||||
}
|
||||
Err(err) => return Err(err),
|
||||
};
|
||||
let multipart_size_limit = multipart_size_limit_from_metadata(&fi.metadata)?;
|
||||
ensure_data_movement_upload_access(&fi, bucket, object, upload_id, opts)?;
|
||||
ensure_multipart_bucket_incarnation(&self.ctx, &fi, bucket, object, upload_id, opts.expected_bucket_incarnation_id)
|
||||
.await?;
|
||||
@@ -1165,6 +1313,44 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
let part_suffix = format!("part.{part_id}");
|
||||
let tmp_part = format!("{}x{}", Uuid::new_v4(), OffsetDateTime::now_utc().unix_timestamp());
|
||||
let tmp_part_path = Arc::new(format!("{tmp_part}/{part_suffix}"));
|
||||
let part_path = format!("{}/{}/{}", upload_id_path, fi.data_dir.unwrap_or_default(), part_suffix);
|
||||
let part_lock_path = format!("{upload_id_path}/{part_suffix}");
|
||||
|
||||
// Keep at most one capped part staging locally per upload. The
|
||||
// distributed lock below is held only for the durable admission check;
|
||||
// it is reacquired for the short final rename, so Complete/Abort are
|
||||
// not blocked behind a slow body upload.
|
||||
let _capped_staging_guard = if multipart_size_limit.is_some() {
|
||||
Some(CappedMultipartStagingGuard {
|
||||
upload_id_path: upload_id_path.clone(),
|
||||
permit: Some(
|
||||
capped_multipart_staging_semaphore(&upload_id_path)
|
||||
.acquire_owned()
|
||||
.await
|
||||
.map_err(|_| Error::other("capped multipart staging semaphore closed"))?,
|
||||
),
|
||||
})
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
if let Some(limit) = multipart_size_limit {
|
||||
let admission_guard = self
|
||||
.acquire_write_lock_diag("put_object_part_admission", RUSTFS_META_MULTIPART_BUCKET, &upload_id_path)
|
||||
.await?;
|
||||
let declared_size = if data.size() >= 0 {
|
||||
u64::try_from(data.size()).map_err(|_| Error::PartMissingOrCorrupt)?
|
||||
} else if data.actual_size() >= 0 {
|
||||
u64::try_from(data.actual_size()).map_err(|_| Error::PartMissingOrCorrupt)?
|
||||
} else {
|
||||
return Err(Error::PartMissingOrCorrupt);
|
||||
};
|
||||
let current_size = self
|
||||
.current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &fi, part_id)
|
||||
.await?;
|
||||
admitted_multipart_size(current_size, declared_size, limit)?;
|
||||
drop(admission_guard);
|
||||
}
|
||||
|
||||
let result: Result<PartInfo> = async {
|
||||
let erasure =
|
||||
@@ -1365,30 +1551,21 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
.await?;
|
||||
}
|
||||
|
||||
let part_path = format!("{}/{}/{}", upload_id_path, fi.data_dir.unwrap_or_default(), part_suffix);
|
||||
let part_lock_path = format!("{upload_id_path}/{part_suffix}");
|
||||
|
||||
#[cfg(test)]
|
||||
pause_multipart_commit(bucket, object, MultipartCommitPause::PutPartBeforeLockAcquire).await;
|
||||
// Serialize only same-part commits (rename_part), not the whole upload.
|
||||
// Each concurrent stream writes to its own unique temp dir (see
|
||||
// `tmp_part` above), so the encode/stream phase never conflicts and must
|
||||
// stay lock-free — holding a lock across it would serialize slow
|
||||
// re-transmits of the same part and defeat the S3 "last finisher wins"
|
||||
// semantics. The mixed-generation hazard is confined to rename_part,
|
||||
// where two temp parts are moved cross-disk onto the SAME final
|
||||
// part_path: interleaving there can leave shards from two generations,
|
||||
// each individually bitrot-valid, that only surface as silent corruption
|
||||
// at read time (backlog#853). A write lock scoped to this part number
|
||||
// makes each same-part commit atomic across disks, so the last committer
|
||||
// wins consistently, while different part numbers commit onto disjoint
|
||||
// part paths and stay concurrent (issue#5961 — an uploadId-wide write
|
||||
// lock serialized them into 503 lock-acquire timeouts). The shared
|
||||
// uploadId read lock keeps completion/abort (which take the uploadId
|
||||
// write lock) from racing any in-flight part commit; a guarded
|
||||
// completion takes the object lock before the upload lock to preserve
|
||||
// global ordering.
|
||||
let (_upload_commit_guard, _part_commit_guard) = if opts.no_lock {
|
||||
// Capped uploads reacquire the upload-wide write lock for the
|
||||
// final durable check and rename. Uncapped uploads retain the
|
||||
// concurrent encode path and only serialize the final same-part
|
||||
// rename; completion/abort use the upload-wide write lock.
|
||||
let (_upload_commit_guard, _part_commit_guard) = if multipart_size_limit.is_some() {
|
||||
let upload_guard = self
|
||||
.acquire_write_lock_diag("put_object_part_commit", RUSTFS_META_MULTIPART_BUCKET, &upload_id_path)
|
||||
.await?;
|
||||
let part_guard = self
|
||||
.acquire_write_lock_diag("put_object_part_commit", RUSTFS_META_MULTIPART_BUCKET, &part_lock_path)
|
||||
.await?;
|
||||
(Some(upload_guard), Some(part_guard))
|
||||
} else if opts.no_lock {
|
||||
(None, None)
|
||||
} else {
|
||||
let upload_guard = self
|
||||
@@ -1400,8 +1577,16 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
(Some(upload_guard), Some(part_guard))
|
||||
};
|
||||
let (commit_fi, _) = self
|
||||
.check_upload_id_exists_with_opts(bucket, object, upload_id, false, opts)
|
||||
.check_upload_id_exists_with_opts(bucket, object, upload_id, multipart_size_limit.is_some(), opts)
|
||||
.await?;
|
||||
let commit_size_limit = multipart_size_limit_from_metadata(&commit_fi.metadata)?;
|
||||
if commit_size_limit != multipart_size_limit {
|
||||
return Err(Error::InvalidArgument(
|
||||
"multipart upload".to_string(),
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
|
||||
"size limit metadata changed or is missing".to_string(),
|
||||
));
|
||||
}
|
||||
ensure_data_movement_upload_access(&commit_fi, bucket, object, upload_id, opts)?;
|
||||
ensure_multipart_bucket_incarnation(
|
||||
&self.ctx,
|
||||
@@ -1431,6 +1616,14 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
}
|
||||
ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
|
||||
|
||||
if let Some(limit) = commit_size_limit {
|
||||
let current_size = self
|
||||
.current_multipart_logical_size(bucket, object, upload_id, &upload_id_path, &commit_fi, part_id)
|
||||
.await?;
|
||||
let candidate_size = u64::try_from(actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
|
||||
admitted_multipart_size(current_size, candidate_size, limit)?;
|
||||
}
|
||||
|
||||
let _ = self
|
||||
.rename_part(
|
||||
&shuffle_disks,
|
||||
@@ -1891,12 +2084,17 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
|
||||
let upload_id_path = Self::get_multipart_upload_dir(bucket, object, upload_id, opts.data_movement);
|
||||
|
||||
self.delete_all_with_quorum(
|
||||
RUSTFS_META_MULTIPART_BUCKET,
|
||||
&upload_id_path,
|
||||
fi.write_quorum(self.default_write_quorum()),
|
||||
)
|
||||
.await
|
||||
let result = self
|
||||
.delete_all_with_quorum(
|
||||
RUSTFS_META_MULTIPART_BUCKET,
|
||||
&upload_id_path,
|
||||
fi.write_quorum(self.default_write_quorum()),
|
||||
)
|
||||
.await;
|
||||
if result.is_ok() {
|
||||
remove_capped_multipart_staging_semaphore(&upload_id_path);
|
||||
}
|
||||
result
|
||||
}
|
||||
// complete_multipart_upload finished
|
||||
#[tracing::instrument(skip(self))]
|
||||
@@ -2016,6 +2214,27 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
return Err(Error::other("part result number err"));
|
||||
}
|
||||
|
||||
if let Some(limit) = multipart_size_limit_from_metadata(&fi.metadata)? {
|
||||
let mut total = 0_u64;
|
||||
for part in &object_parts {
|
||||
if part.error.is_some() {
|
||||
return Err(Error::PartMissingOrCorrupt);
|
||||
}
|
||||
let part_size = u64::try_from(part.actual_size).map_err(|_| Error::PartMissingOrCorrupt)?;
|
||||
total = total.checked_add(part_size).ok_or_else(|| {
|
||||
Error::InvalidArgument(
|
||||
"multipart upload".to_string(),
|
||||
SUFFIX_MAX_TOTAL_OBJECT_SIZE.to_string(),
|
||||
"logical size overflow".to_string(),
|
||||
)
|
||||
})?;
|
||||
}
|
||||
if total > limit {
|
||||
return Err(Error::EntityTooLarge(total, limit));
|
||||
}
|
||||
rustfs_utils::http::metadata_compat::remove_str(&mut fi.metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE);
|
||||
}
|
||||
|
||||
let mut checksum_type = rustfs_rio::ChecksumType::NONE;
|
||||
|
||||
if let Some(cs) = fi.metadata.get(rustfs_rio::RUSTFS_MULTIPART_CHECKSUM) {
|
||||
@@ -3024,13 +3243,17 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
Ok(ObjectInfo::from_file_info(&fi, &commit_bucket, &commit_object, commit_is_versioned))
|
||||
};
|
||||
|
||||
if detach_commit_owner {
|
||||
let result = if detach_commit_owner {
|
||||
tokio::spawn(commit)
|
||||
.await
|
||||
.map_err(|err| Error::other(format!("complete_multipart_upload commit task failed: {err}")))?
|
||||
} else {
|
||||
commit.await
|
||||
};
|
||||
if result.is_ok() {
|
||||
remove_capped_multipart_staging_semaphore(&upload_id_path);
|
||||
}
|
||||
result
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3096,6 +3319,34 @@ mod tests {
|
||||
assert!(multipart_bucket_incarnation_id(&nil_metadata).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multipart_size_limit_metadata_is_dual_key_and_fail_closed() {
|
||||
let mut metadata = HashMap::new();
|
||||
insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "100".to_string());
|
||||
assert_eq!(multipart_size_limit_from_metadata(&metadata).unwrap(), Some(100));
|
||||
|
||||
metadata.insert("x-minio-internal-max-total-object-size".to_string(), "101".to_string());
|
||||
assert!(multipart_size_limit_from_metadata(&metadata).is_err());
|
||||
|
||||
let mut invalid = HashMap::new();
|
||||
insert_str(&mut invalid, SUFFIX_MAX_TOTAL_OBJECT_SIZE, "-1".to_string());
|
||||
assert!(multipart_size_limit_from_metadata(&invalid).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multipart_size_admission_handles_boundaries_and_overflow() {
|
||||
assert_eq!(admitted_multipart_size(90, 10, 100).unwrap(), 100);
|
||||
assert!(matches!(
|
||||
admitted_multipart_size(90, 11, 100),
|
||||
Err(StorageError::EntityTooLarge(101, 100))
|
||||
));
|
||||
assert!(admitted_multipart_size(u64::MAX - 1, 1, u64::MAX).is_ok());
|
||||
assert!(matches!(
|
||||
admitted_multipart_size(u64::MAX, 1, u64::MAX),
|
||||
Err(StorageError::InvalidArgument(_, _, _))
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multipart_bucket_incarnation_gate_accepts_only_current_or_same_lifetime_legacy_uploads() {
|
||||
let expected = Uuid::new_v4();
|
||||
|
||||
@@ -30,31 +30,34 @@ use super::super::{
|
||||
GET_OBJECT_PATH_DIRECT_MEMORY, GET_OBJECT_PATH_EMPTY, GET_OBJECT_PATH_INLINE_DIRECT, GET_OBJECT_PATH_INTERNAL_META,
|
||||
GET_OBJECT_PATH_LEGACY_DUPLEX, GET_OBJECT_PATH_REMOTE_TRANSITION, GET_OBJECT_PATH_SET_DISK, GET_STAGE_DECODE, GET_STAGE_EMIT,
|
||||
GET_STAGE_INLINE_PREPARE, GET_STAGE_LOCK_ACQUIRE, GET_STAGE_METADATA, GET_STAGE_OBJECT_INFO, GET_STAGE_PATH_DECISION,
|
||||
GET_STAGE_READER_SETUP, GenericError, GetCodecStreamingDecision, GetDirectMemoryDecision, GetObjectReader, HTTPRangeSpec,
|
||||
HashAlgorithm, HashMap, HashReader, HashSet, HeaderMap, HealChannelPriority, InstanceContext, Instant, LOG_COMPONENT_ECSTORE,
|
||||
LOG_SUBSYSTEM_SET_DISK, OBJECT_OP_IGNORED_ERRS, ObjectApiError, ObjectInfo, ObjectKey, ObjectLockConfigSnapshot,
|
||||
ObjectLockConfigState, ObjectOptions, ObjectReader, ObjectToDelete, OffsetDateTime, Ordering, Pin, PutObjReader,
|
||||
RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET, ReaderImpl, ReplicateDecision, ReplicationObjectBridge, Result,
|
||||
SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS, SLASH_SEPARATOR, SUFFIX_ACTUAL_SIZE, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE,
|
||||
SUFFIX_RESTORE_OPERATION_ID, SetDisks, SmallWritePath, StorageError, TRANSITION_COMPLETE, UpdateMetadataOpts, Uuid,
|
||||
WriteLayout, X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, X_AMZ_RESTORE,
|
||||
adaptive_duplex_buffer_size, build_get_object_info, build_inline_bitrot_readers, build_inline_bitrot_readers_from_refs,
|
||||
can_try_inline_data_shards_direct, check_object_lock_delete, check_object_lock_for_deletion_with_state,
|
||||
check_object_lock_retention_update, classify_get_codec_streaming_object_class, classify_put_write_path,
|
||||
classify_storage_error, collect_inline_data_shard_fileinfos_by_index, contains_key_str, create_bitrot_writer, debug,
|
||||
delete_file_info_version_id, disk, ensure_delete_commit_locks_held, error, finish_set_disk_read_lock,
|
||||
get_codec_streaming_reader_gate, get_object_body_cache_hook, get_raw_etag, get_small_object_direct_memory_decision,
|
||||
GET_STAGE_READER_SETUP, GenericError, GetCodecStreamingDecision, GetCodecStreamingFallbackReason, GetDirectMemoryDecision,
|
||||
GetObjectReader, HTTPRangeSpec, HashAlgorithm, HashMap, HashReader, HashSet, HeaderMap, HealChannelPriority, InstanceContext,
|
||||
Instant, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_SET_DISK, OBJECT_OP_IGNORED_ERRS, ObjectApiError, ObjectInfo, ObjectKey,
|
||||
ObjectLockConfigSnapshot, ObjectLockConfigState, ObjectOptions, ObjectReader, ObjectToDelete, OffsetDateTime, Ordering, Pin,
|
||||
PutObjReader, RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET, ReadPathPlan, ReaderImpl, ReplicateDecision,
|
||||
ReplicationObjectBridge, Result, SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS, SLASH_SEPARATOR, SUFFIX_ACTUAL_SIZE,
|
||||
SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_RESTORE_OPERATION_ID, SetDisks, SmallWritePath, StorageError,
|
||||
TRANSITION_COMPLETE, UpdateMetadataOpts, Uuid, WriteLayout, X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE,
|
||||
X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, X_AMZ_RESTORE, adaptive_duplex_buffer_size, build_get_object_info,
|
||||
build_inline_bitrot_readers, build_inline_bitrot_readers_from_refs, can_try_inline_data_shards_direct,
|
||||
check_object_lock_delete, check_object_lock_for_deletion_with_state, check_object_lock_retention_update,
|
||||
classify_get_codec_streaming_object_class, classify_put_write_path, classify_storage_error,
|
||||
collect_inline_data_shard_fileinfos_by_index, contains_key_str, create_bitrot_writer, debug, delete_file_info_version_id,
|
||||
disk, ensure_delete_commit_locks_held, error, explicit_delete_removed_marker, finish_set_disk_read_lock,
|
||||
get_codec_streaming_reader_gate_with_plan, get_object_body_cache_hook, get_raw_etag,
|
||||
get_small_object_direct_memory_decision_with_threshold_and_plan, get_small_object_direct_memory_threshold,
|
||||
get_stage_timer_if_enabled, get_str, get_transitioned_object_reader_with_tier_manager, inline_erasure_shard_file_offset,
|
||||
inline_erasure_shard_size, insert_str, is_deadlock_detection_enabled, is_err_object_not_found, is_err_version_not_found,
|
||||
is_explicit_null_version, is_lock_optimization_enabled, issue3031_diag_enabled, join_all, known_put_object_storage_size,
|
||||
path_join_buf, put_restore_opts, record_compression_total_memory, record_get_codec_streaming_gate_decision,
|
||||
is_explicit_null_version, is_get_codec_streaming_base_enabled, is_get_small_object_direct_memory_enabled,
|
||||
is_lock_optimization_enabled, issue3031_diag_enabled, join_all, known_put_object_storage_size, path_join_buf,
|
||||
put_restore_opts, record_compression_total_memory, record_get_codec_streaming_gate_decision,
|
||||
record_get_direct_memory_decision, record_get_object_pipeline_failure, record_get_object_pipeline_failure_for_path,
|
||||
record_get_object_reader_path_observation, record_get_stage_duration_if_enabled, record_lock_acquire,
|
||||
reduce_write_quorum_errs, release_materialized_read_lock, replication_write_may_pass_worm_gate, require_restore_operation_id,
|
||||
resolve_delete_version_state, resolve_tiered_decommission_write_quorum_result, resolve_write_layout,
|
||||
restore_commit_operation_id_from_metadata, restore_operation_id_from_metadata, send_event,
|
||||
set_disk_delete_creates_delete_marker, should_force_delete_marker_for_missing_version,
|
||||
should_persist_encryption_original_size, should_preserve_delete_replication_state, should_use_inline_fast_path,
|
||||
should_persist_encryption_original_size, should_preserve_delete_replication_state, should_use_inline_fast_path_with_plan,
|
||||
take_prepared_get_object_metadata, to_object_err, try_read_inline_data_shards_direct, warn,
|
||||
};
|
||||
use super::bitrot_self_verify::{BitrotSelfVerifyTarget, drop_failed_writer_disks, verify_written_bitrot_shards};
|
||||
@@ -67,7 +70,7 @@ use crate::set_disk::coding;
|
||||
use crate::set_disk::core::io_primitives::GetCodecStreamingReaderBuildOutcome;
|
||||
use crate::set_disk::mem;
|
||||
use crate::set_disk::metadata_sys;
|
||||
use crate::set_disk::read::GetObjectDownstreamWriter;
|
||||
use crate::set_disk::read::{GET_OBJECT_PATH_MID_SIZE_STREAMING, GetObjectDownstreamWriter};
|
||||
use crate::set_disk::runtime_sources;
|
||||
use crate::storage_api_contracts::multipart::MultipartOperations;
|
||||
use crate::storage_api_contracts::object::ObjectIO;
|
||||
@@ -79,6 +82,111 @@ use rustfs_rio::TryGetIndex;
|
||||
use rustfs_utils::http::HeaderExt;
|
||||
use tokio::io::AsyncWriteExt;
|
||||
|
||||
// Keep the mid-size reader separate from the codec-streaming rollout. The
|
||||
// latter is intentionally opt-in because its per-stripe worker can regress
|
||||
// tiny objects; this bounded range is the gap between direct-memory GETs and
|
||||
// the legacy duplex reader.
|
||||
const ENV_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE: &str = "RUSTFS_GET_MID_SIZE_STREAMING_ENABLE";
|
||||
const DEFAULT_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE: bool = true;
|
||||
const GET_MID_SIZE_STREAMING_MIN_SIZE: usize = 128 * 1024 + 1;
|
||||
const GET_MID_SIZE_STREAMING_MAX_SIZE: usize = 1024 * 1024;
|
||||
|
||||
fn is_get_mid_size_streaming_enabled() -> bool {
|
||||
#[cfg(test)]
|
||||
{
|
||||
rustfs_utils::get_env_bool(ENV_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE, DEFAULT_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE)
|
||||
}
|
||||
#[cfg(not(test))]
|
||||
{
|
||||
static ENABLED: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
|
||||
*ENABLED.get_or_init(|| {
|
||||
rustfs_utils::get_env_bool(ENV_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE, DEFAULT_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Return the object size when the bounded non-duplex reader is safe.
|
||||
///
|
||||
/// This predicate deliberately has a narrower contract than the general
|
||||
/// codec-streaming gate: only a whole, plain, single-part object is eligible.
|
||||
/// Ranges, transforms, remote objects, multipart reads, copy-source reads and
|
||||
/// special movement/version requests retain their existing legacy semantics.
|
||||
#[allow(dead_code, reason = "asserted by this file's gate tests")]
|
||||
fn get_mid_size_streaming_object_size(
|
||||
range: &Option<HTTPRangeSpec>,
|
||||
object_info: &ObjectInfo,
|
||||
fi: &FileInfo,
|
||||
opts: &ObjectOptions,
|
||||
lock_optimization_enabled: bool,
|
||||
) -> Option<usize> {
|
||||
get_mid_size_streaming_object_size_with_flags(
|
||||
range,
|
||||
object_info,
|
||||
fi,
|
||||
opts,
|
||||
lock_optimization_enabled,
|
||||
is_get_mid_size_streaming_enabled(),
|
||||
is_get_codec_streaming_base_enabled(),
|
||||
)
|
||||
}
|
||||
|
||||
#[allow(dead_code, reason = "asserted by this file's gate tests")]
|
||||
fn get_mid_size_streaming_object_size_with_flags(
|
||||
range: &Option<HTTPRangeSpec>,
|
||||
object_info: &ObjectInfo,
|
||||
fi: &FileInfo,
|
||||
opts: &ObjectOptions,
|
||||
lock_optimization_enabled: bool,
|
||||
mid_size_enabled: bool,
|
||||
codec_base_enabled: bool,
|
||||
) -> Option<usize> {
|
||||
get_mid_size_streaming_object_size_with_flags_and_plan(
|
||||
range,
|
||||
object_info,
|
||||
opts,
|
||||
lock_optimization_enabled,
|
||||
mid_size_enabled,
|
||||
codec_base_enabled,
|
||||
super::super::ReadPathPlan::new(object_info, fi),
|
||||
)
|
||||
}
|
||||
|
||||
fn get_mid_size_streaming_object_size_with_flags_and_plan(
|
||||
range: &Option<HTTPRangeSpec>,
|
||||
object_info: &ObjectInfo,
|
||||
opts: &ObjectOptions,
|
||||
lock_optimization_enabled: bool,
|
||||
mid_size_enabled: bool,
|
||||
codec_base_enabled: bool,
|
||||
plan: super::super::ReadPathPlan,
|
||||
) -> Option<usize> {
|
||||
if !mid_size_enabled
|
||||
|| !codec_base_enabled
|
||||
|| !lock_optimization_enabled
|
||||
|| range.is_some()
|
||||
|| opts.part_number.is_some()
|
||||
|| opts.version_id.is_some()
|
||||
|| opts.incl_free_versions
|
||||
|| opts.skip_free_version
|
||||
|| opts.data_movement
|
||||
|| opts.raw_data_movement_read
|
||||
|| object_info.delete_marker
|
||||
|| object_info.metadata_only
|
||||
|| object_info.version_only
|
||||
|| crate::set_disk::get_object_read_policy() != super::super::GetObjectReadPolicy::Default
|
||||
{
|
||||
return None;
|
||||
}
|
||||
|
||||
let shape = plan.shape()?;
|
||||
if !plan.is_plain() {
|
||||
return None;
|
||||
}
|
||||
(GET_MID_SIZE_STREAMING_MIN_SIZE..=GET_MID_SIZE_STREAMING_MAX_SIZE)
|
||||
.contains(&shape.object_size)
|
||||
.then_some(shape.object_size)
|
||||
}
|
||||
|
||||
#[cfg(all(test, feature = "test-util"))]
|
||||
use super::super::GetObjectMetadataCacheEntry;
|
||||
#[cfg(test)]
|
||||
@@ -1629,11 +1737,21 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
|
||||
return Ok(reader);
|
||||
}
|
||||
|
||||
// All remaining local fast paths share this immutable, fail-closed
|
||||
// metadata decision. Build it once after empty/remote exits so those
|
||||
// requests do not pay for part and transform scans they cannot use.
|
||||
let read_path_plan = ReadPathPlan::new(&object_info, fi);
|
||||
|
||||
// Inline data fast path: skip duplex pipe for small inline objects.
|
||||
// Uses the shared predicate from ObjectInfo; additionally checks that
|
||||
// inline data is actually present and neither range nor partNumber is
|
||||
// in flight.
|
||||
if should_use_inline_fast_path(&range, &object_info, fi, opts) {
|
||||
let use_inline_fast_path = object_info.is_inline_fast_path_eligible()
|
||||
&& fi.data.is_some()
|
||||
&& range.is_none()
|
||||
&& opts.part_number.is_none()
|
||||
&& should_use_inline_fast_path_with_plan(&range, &object_info, fi, opts, read_path_plan);
|
||||
if use_inline_fast_path {
|
||||
let mut inline_prepare_stage_start = get_stage_timer_if_enabled(stage_metrics_enabled);
|
||||
let data_shards = fi.erasure.data_blocks;
|
||||
|
||||
@@ -1795,27 +1913,10 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
|
||||
}
|
||||
}
|
||||
|
||||
let path_decision_stage_start = get_stage_timer_if_enabled(stage_metrics_enabled);
|
||||
let codec_streaming_gate = get_codec_streaming_reader_gate(
|
||||
bucket,
|
||||
object,
|
||||
opts.part_number,
|
||||
object_class,
|
||||
&object_info,
|
||||
fi,
|
||||
lock_optimization_enabled,
|
||||
);
|
||||
record_get_stage_duration_if_enabled(GET_OBJECT_PATH_SET_DISK, GET_STAGE_PATH_DECISION, path_decision_stage_start);
|
||||
|
||||
if object_info.is_remote() {
|
||||
if let GetCodecStreamingDecision::Fallback(reason) = codec_streaming_gate.decision {
|
||||
record_get_codec_streaming_gate_decision(
|
||||
codec_streaming_gate.object_class,
|
||||
codec_streaming_gate.decision,
|
||||
size_bucket,
|
||||
);
|
||||
rustfs_io_metrics::record_get_object_codec_streaming_fallback(reason.as_str());
|
||||
}
|
||||
let decision = GetCodecStreamingDecision::Fallback(GetCodecStreamingFallbackReason::Remote);
|
||||
record_get_codec_streaming_gate_decision(object_class, decision, size_bucket);
|
||||
rustfs_io_metrics::record_get_object_codec_streaming_fallback(GetCodecStreamingFallbackReason::Remote.as_str());
|
||||
record_get_object_reader_path_observation(GET_OBJECT_PATH_REMOTE_TRANSITION, object_class, size_bucket);
|
||||
let mut opts = opts.clone();
|
||||
if object_info.parts.len() == 1 {
|
||||
@@ -1835,6 +1936,10 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
|
||||
return Ok(finish_set_disk_read_lock(gr, read_lock_guard.take(), bucket, object));
|
||||
}
|
||||
|
||||
// Metadata resolution and the remote-tier branch are complete here.
|
||||
// Keep the rollout/configuration gate deferred until the request
|
||||
// really needs codec streaming so an opted-out codec path cannot add
|
||||
// fixed cost to the inline/direct-memory/mid-size hot paths.
|
||||
// App-layer object data cache probe: metadata (etag/size) is resolved
|
||||
// but no data shards have been read yet, so a hit skips the erasure
|
||||
// read, bitrot verify and decode entirely. The hook validates object
|
||||
@@ -1881,7 +1986,15 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
|
||||
}
|
||||
}
|
||||
|
||||
let direct_memory_decision = get_small_object_direct_memory_decision(&range, &object_info, fi, opts);
|
||||
let direct_memory_decision = get_small_object_direct_memory_decision_with_threshold_and_plan(
|
||||
&range,
|
||||
&object_info,
|
||||
fi,
|
||||
opts,
|
||||
is_get_small_object_direct_memory_enabled(),
|
||||
get_small_object_direct_memory_threshold(),
|
||||
read_path_plan,
|
||||
);
|
||||
record_get_direct_memory_decision(object_class, direct_memory_decision, size_bucket);
|
||||
if let GetDirectMemoryDecision::Use { object_size } = direct_memory_decision {
|
||||
if let Some(body) = Self::try_get_object_direct_data_shards_with_fileinfo(
|
||||
@@ -1962,6 +2075,66 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
|
||||
return Ok(reader);
|
||||
}
|
||||
|
||||
// Mid-size plain objects use a bounded, non-duplex reader. Keep this
|
||||
// path independent from the general codec-streaming rollout: that
|
||||
// rollout remains off by default because its worker overhead is not a
|
||||
// win for tiny objects. A failed setup degrades to the existing codec
|
||||
// gate/legacy path before any response bytes are returned.
|
||||
if get_mid_size_streaming_object_size_with_flags_and_plan(
|
||||
&range,
|
||||
&object_info,
|
||||
opts,
|
||||
lock_optimization_enabled,
|
||||
is_get_mid_size_streaming_enabled(),
|
||||
is_get_codec_streaming_base_enabled(),
|
||||
read_path_plan,
|
||||
)
|
||||
.is_some()
|
||||
{
|
||||
match Self::get_object_mid_size_reader_with_fileinfo(
|
||||
bucket,
|
||||
object,
|
||||
Arc::clone(&self.erasure_cache),
|
||||
fi,
|
||||
files,
|
||||
disks,
|
||||
self.set_index,
|
||||
self.pool_index,
|
||||
opts.skip_verify_bitrot,
|
||||
object_class.as_str(),
|
||||
size_bucket,
|
||||
false,
|
||||
)
|
||||
.await?
|
||||
{
|
||||
GetCodecStreamingReaderBuildOutcome::Reader(stream) => {
|
||||
record_get_object_reader_path_observation(GET_OBJECT_PATH_MID_SIZE_STREAMING, object_class, size_bucket);
|
||||
let (mut reader, _offset, _length) =
|
||||
get_object_reader_with_context(&self.ctx, stream, range, &object_info, opts, &h).await?;
|
||||
reader.body_source = body_source;
|
||||
return Ok(finish_set_disk_read_lock(reader, read_lock_guard.take(), bucket, object));
|
||||
}
|
||||
GetCodecStreamingReaderBuildOutcome::Fallback(_) => {
|
||||
// The setup found a degraded-but-readable layout. Let the
|
||||
// established codec gate and then legacy path handle it;
|
||||
// this preserves whole-request fallback semantics.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let path_decision_stage_start = get_stage_timer_if_enabled(stage_metrics_enabled);
|
||||
let codec_streaming_gate = get_codec_streaming_reader_gate_with_plan(
|
||||
bucket,
|
||||
object,
|
||||
opts.part_number,
|
||||
object_class,
|
||||
&object_info,
|
||||
fi,
|
||||
lock_optimization_enabled,
|
||||
read_path_plan,
|
||||
);
|
||||
record_get_stage_duration_if_enabled(GET_OBJECT_PATH_SET_DISK, GET_STAGE_PATH_DECISION, path_decision_stage_start);
|
||||
|
||||
match codec_streaming_gate.decision {
|
||||
GetCodecStreamingDecision::Use => {
|
||||
match Self::get_object_decode_reader_with_fileinfo(
|
||||
@@ -2458,29 +2631,19 @@ impl SetDisks {
|
||||
|
||||
let mut object_lock_guard = None;
|
||||
let mut bucket_lifecycle_guard = None;
|
||||
let deferred_data_movement_precondition = opts.data_movement && opts.http_preconditions.is_some();
|
||||
|
||||
if opts.http_preconditions.is_some() && !deferred_data_movement_precondition {
|
||||
if !opts.no_lock {
|
||||
if let Some(expected_incarnation_id) = opts.expected_bucket_incarnation_id
|
||||
&& opts.bucket_lifecycle_lock_fence.is_none()
|
||||
{
|
||||
bucket_lifecycle_guard = Some(
|
||||
metadata_sys::object_store_in(&self.ctx)
|
||||
.await?
|
||||
.acquire_bucket_incarnation_fence(bucket, expected_incarnation_id)
|
||||
.await?,
|
||||
);
|
||||
}
|
||||
object_lock_guard = Some(
|
||||
self.acquire_write_lock_diag("put_object_precondition", bucket, object)
|
||||
.await?,
|
||||
);
|
||||
}
|
||||
|
||||
if let Some(err) = self.check_write_precondition(bucket, object, opts).await {
|
||||
return Err(err);
|
||||
}
|
||||
// This pre-body check is advisory fast-fail only: the authoritative
|
||||
// precondition evaluation happens under the commit namespace lock
|
||||
// below, so the namespace write lock must NOT be taken here — holding
|
||||
// it across client-paced body ingestion starves concurrent reads of
|
||||
// the same object into lock-timeout 503s (rustfs/backlog#2074).
|
||||
// Data movement skips the advisory read: its staleness predicate is
|
||||
// only meaningful at commit time.
|
||||
if opts.http_preconditions.is_some()
|
||||
&& !opts.data_movement
|
||||
&& let Some(err) = self.check_write_precondition(bucket, object, opts).await
|
||||
{
|
||||
return Err(err);
|
||||
}
|
||||
|
||||
let expected_restore_operation_id = restore_commit_operation_id_from_metadata(&opts.user_defined)?;
|
||||
@@ -2950,7 +3113,9 @@ impl SetDisks {
|
||||
#[cfg(any(test, feature = "test-util"))]
|
||||
pause_put_object_commit(bucket, object, PutObjectCommitPause::AfterNamespace).await;
|
||||
|
||||
if deferred_data_movement_precondition && let Some(err) = self.check_write_precondition(bucket, object, opts).await {
|
||||
if opts.http_preconditions.is_some()
|
||||
&& let Some(err) = self.check_write_precondition(bucket, object, opts).await
|
||||
{
|
||||
return Err(err);
|
||||
}
|
||||
|
||||
@@ -6521,7 +6686,21 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
}
|
||||
}
|
||||
|
||||
if goi.delete_marker && dobj.version_id.is_some() && goi.version_id == version_id {
|
||||
// Same normalization as `explicit_delete_marker` above: `goi.version_id`
|
||||
// is the client-facing identity (`Some(Uuid::nil())` for a null
|
||||
// version) while `version_id` is the storage identity (`None` for an
|
||||
// explicit null). Comparing them raw made a null delete marker's
|
||||
// removal take the non-marker branch below, so the response lost
|
||||
// `DeleteMarker`/`DeleteMarkerVersionId` and the removal was
|
||||
// accounted as an object deletion (issue #6745).
|
||||
let removed_delete_marker =
|
||||
goi.delete_marker && dobj.version_id.is_some() && delete_file_info_version_id(goi.version_id) == version_id;
|
||||
// Response-side only for the null identity: a delete request marked
|
||||
// `deleted` with `version_id == None` makes `FileMeta::delete_version`
|
||||
// re-create the marker it just removed (the suspended-bucket
|
||||
// "delete mints a marker" write path), so the write shape must stay
|
||||
// untouched for explicit null-marker removals.
|
||||
if removed_delete_marker && version_id.is_some() {
|
||||
vr.deleted = true;
|
||||
vr.mod_time = goi.mod_time;
|
||||
}
|
||||
@@ -6540,11 +6719,21 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
}
|
||||
};
|
||||
|
||||
if vr.deleted {
|
||||
if vr.deleted || removed_delete_marker {
|
||||
del_objects[i] = DeletedObject {
|
||||
delete_marker: vr.deleted,
|
||||
delete_marker_version_id: vr.version_id,
|
||||
delete_marker_mtime: vr.mod_time,
|
||||
delete_marker: true,
|
||||
// `vr.version_id` holds the storage identity, which is
|
||||
// `None` for an explicit null-marker removal; report the
|
||||
// client-facing null identity so the response can carry
|
||||
// `DeleteMarkerVersionId` for the marker that was removed.
|
||||
delete_marker_version_id: if explicit_null_version {
|
||||
Some(Uuid::nil())
|
||||
} else {
|
||||
vr.version_id
|
||||
},
|
||||
// For a null-marker removal `vr` stays undeleted (write
|
||||
// shape), so take the marker's mtime from the source.
|
||||
delete_marker_mtime: vr.mod_time.or(goi.mod_time),
|
||||
object_name: vr.name.clone(),
|
||||
replication_state: vr.replication_state_internal.clone(),
|
||||
..Default::default()
|
||||
@@ -7197,6 +7386,14 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
obj_info.user_defined = Arc::clone(&goi.user_defined);
|
||||
obj_info.parts = Arc::clone(&goi.parts);
|
||||
obj_info.user_tags = Arc::clone(&goi.user_tags);
|
||||
// Report delete-marker semantics for an explicit-version delete whose
|
||||
// target was a delete marker. On versioning-suspended buckets
|
||||
// `resolve_delete_version_state` cannot mark the write request itself
|
||||
// (a `deleted` write with the null identity re-creates the marker), so
|
||||
// the marker-ness is restored on the response here (issue #6745).
|
||||
if explicit_delete_removed_marker(&opts, &goi, version_found) {
|
||||
obj_info.delete_marker = true;
|
||||
}
|
||||
self.invalidate_get_object_metadata_cache(bucket, object).await;
|
||||
Ok(obj_info)
|
||||
}
|
||||
@@ -8165,6 +8362,190 @@ mod erasure_construction_tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod mid_size_streaming_gate_tests {
|
||||
use super::*;
|
||||
use rustfs_filemeta::ObjectPartInfo;
|
||||
use serial_test::serial;
|
||||
|
||||
fn plain_metadata(size: usize) -> (ObjectInfo, FileInfo) {
|
||||
let size_i64 = i64::try_from(size).expect("test size fits i64");
|
||||
let mut fi = FileInfo::new("object", 2, 2);
|
||||
fi.size = size_i64;
|
||||
fi.add_object_part(1, "etag".to_string(), size, fi.mod_time, size_i64, None, None);
|
||||
let object_info = ObjectInfo {
|
||||
size: size_i64,
|
||||
parts: Arc::new(vec![ObjectPartInfo {
|
||||
number: 1,
|
||||
size,
|
||||
actual_size: size_i64,
|
||||
..Default::default()
|
||||
}]),
|
||||
..Default::default()
|
||||
};
|
||||
(object_info, fi)
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn direct_memory_boundary_stays_out_of_mid_size_streaming() {
|
||||
let (object_info, fi) = plain_metadata(128 * 1024);
|
||||
assert_eq!(
|
||||
get_mid_size_streaming_object_size_with_flags(&None, &object_info, &fi, &ObjectOptions::default(), true, true, true),
|
||||
None
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn mid_size_streaming_accepts_object_just_above_direct_memory_ceiling() {
|
||||
let (object_info, fi) = plain_metadata(128 * 1024 + 1);
|
||||
assert_eq!(
|
||||
get_mid_size_streaming_object_size_with_flags(&None, &object_info, &fi, &ObjectOptions::default(), true, true, true),
|
||||
Some(128 * 1024 + 1)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn mid_size_streaming_includes_one_mib_and_rejects_larger_objects() {
|
||||
let (object_info, fi) = plain_metadata(1024 * 1024);
|
||||
assert_eq!(
|
||||
get_mid_size_streaming_object_size_with_flags(&None, &object_info, &fi, &ObjectOptions::default(), true, true, true),
|
||||
Some(1024 * 1024)
|
||||
);
|
||||
|
||||
let (large_info, large_fi) = plain_metadata(1024 * 1024 + 1);
|
||||
assert_eq!(
|
||||
get_mid_size_streaming_object_size_with_flags(
|
||||
&None,
|
||||
&large_info,
|
||||
&large_fi,
|
||||
&ObjectOptions::default(),
|
||||
true,
|
||||
true,
|
||||
true
|
||||
),
|
||||
None
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn mid_size_streaming_rejects_ranges_and_transformed_objects() {
|
||||
let (object_info, fi) = plain_metadata(256 * 1024);
|
||||
let range = Some(HTTPRangeSpec {
|
||||
is_suffix_length: false,
|
||||
start: 0,
|
||||
end: 1,
|
||||
});
|
||||
assert_eq!(
|
||||
get_mid_size_streaming_object_size_with_flags(&range, &object_info, &fi, &ObjectOptions::default(), true, true, true),
|
||||
None
|
||||
);
|
||||
|
||||
let mut encrypted = object_info;
|
||||
encrypted.user_defined = Arc::new(HashMap::from([("x-minio-encryption-key".to_string(), "opaque".to_string())]));
|
||||
assert_eq!(
|
||||
get_mid_size_streaming_object_size_with_flags(&None, &encrypted, &fi, &ObjectOptions::default(), true, true, true),
|
||||
None
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn mid_size_streaming_rejects_inconsistent_part_geometry() {
|
||||
let (object_info, mut fi) = plain_metadata(256 * 1024);
|
||||
fi.parts[0].size += 1;
|
||||
assert_eq!(
|
||||
get_mid_size_streaming_object_size_with_flags(&None, &object_info, &fi, &ObjectOptions::default(), true, true, true),
|
||||
None
|
||||
);
|
||||
|
||||
let (mut object_info, fi) = plain_metadata(256 * 1024);
|
||||
object_info.parts = Arc::new(vec![ObjectPartInfo {
|
||||
number: 2,
|
||||
size: fi.parts[0].size,
|
||||
actual_size: fi.parts[0].actual_size,
|
||||
..Default::default()
|
||||
}]);
|
||||
assert_eq!(
|
||||
get_mid_size_streaming_object_size_with_flags(&None, &object_info, &fi, &ObjectOptions::default(), true, true, true),
|
||||
None
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn mid_size_streaming_can_be_disabled_without_affecting_direct_memory_gate() {
|
||||
temp_env::with_vars(
|
||||
[
|
||||
(ENV_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE, Some("false")),
|
||||
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_ENABLE, Some("true")),
|
||||
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED, Some("true")),
|
||||
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED, Some("true")),
|
||||
],
|
||||
|| {
|
||||
let (object_info, fi) = plain_metadata(256 * 1024);
|
||||
assert_eq!(
|
||||
get_mid_size_streaming_object_size(&None, &object_info, &fi, &ObjectOptions::default(), true),
|
||||
None
|
||||
);
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn mid_size_streaming_respects_codec_compatibility_kill_switches() {
|
||||
let (object_info, fi) = plain_metadata(256 * 1024);
|
||||
for (name, value) in [
|
||||
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_ENABLE, "false"),
|
||||
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED, "false"),
|
||||
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED, "false"),
|
||||
] {
|
||||
temp_env::with_vars(
|
||||
[
|
||||
(ENV_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE, Some("true")),
|
||||
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_ENABLE, Some("true")),
|
||||
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED, Some("true")),
|
||||
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED, Some("true")),
|
||||
(name, Some(value)),
|
||||
],
|
||||
|| {
|
||||
assert_eq!(
|
||||
get_mid_size_streaming_object_size(&None, &object_info, &fi, &ObjectOptions::default(), true),
|
||||
None
|
||||
);
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn mid_size_streaming_rejects_copy_source_policy() {
|
||||
let (object_info, fi) = plain_metadata(256 * 1024);
|
||||
let result = tokio::runtime::Runtime::new()
|
||||
.expect("test runtime should initialize")
|
||||
.block_on(crate::set_disk::with_get_object_read_policy(
|
||||
crate::set_disk::GetObjectReadPolicy::CopySource,
|
||||
async {
|
||||
get_mid_size_streaming_object_size_with_flags(
|
||||
&None,
|
||||
&object_info,
|
||||
&fi,
|
||||
&ObjectOptions::default(),
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
)
|
||||
},
|
||||
));
|
||||
assert_eq!(result, None);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod object_encryption_resolver_wiring_tests {
|
||||
use super::*;
|
||||
@@ -9297,7 +9678,7 @@ mod replication_lww_tests {
|
||||
mod inline_put_commit_path_tests {
|
||||
use super::hermetic_set_disks_support::hermetic_set_disks_isolated as hermetic_set_disks;
|
||||
use super::*;
|
||||
use crate::config::storageclass::lookup_config_for_pools_without_env;
|
||||
use crate::config::storageclass::{INLINE_BLOCK_ENV, lookup_config_for_pools, lookup_config_for_pools_without_env};
|
||||
use crate::disk::ReadOptions;
|
||||
use rustfs_config::server_config::KVS;
|
||||
use serial_test::serial;
|
||||
@@ -9364,6 +9745,62 @@ mod inline_put_commit_path_tests {
|
||||
assert_eq!(restored, payload);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn get_object_reader_wires_mid_size_to_single_inflight() {
|
||||
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
|
||||
let bucket = "mid-size-reader-wiring";
|
||||
let object = "object.bin";
|
||||
let payload: Vec<u8> = (0..256 * 1024).map(|index| (index % 251) as u8).collect();
|
||||
make_bucket(&disk_stores, bucket).await;
|
||||
let storage_class = temp_env::with_var(INLINE_BLOCK_ENV, Some("1KiB"), || lookup_config_for_pools(&KVS::new(), &[4]))
|
||||
.expect("test storage class should resolve");
|
||||
set_disks.set_test_storage_class_config(storage_class);
|
||||
|
||||
let mut writer = PutObjReader::from_vec(payload.clone());
|
||||
temp_env::async_with_vars(
|
||||
[
|
||||
(ENV_RUSTFS_GET_MID_SIZE_STREAMING_ENABLE, Some("true")),
|
||||
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_ENABLE, Some("true")),
|
||||
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED, Some("true")),
|
||||
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED, Some("true")),
|
||||
(crate::set_disk::ENV_RUSTFS_GET_CODEC_STREAMING_ROLLOUT, Some("on")),
|
||||
(rustfs_config::ENV_OBJECT_LOCK_OPTIMIZATION_ENABLE, Some("true")),
|
||||
],
|
||||
async {
|
||||
set_disks
|
||||
.put_object(bucket, object, &mut writer, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("mid-size wiring fixture should commit");
|
||||
|
||||
crate::set_disk::reset_test_get_object_reader_path();
|
||||
let single_inflight_before = crate::set_disk::coding::decode_reader::test_single_inflight_construction_count();
|
||||
let mut reader = set_disks
|
||||
.get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
|
||||
.await
|
||||
.expect("mid-size wiring GET should succeed");
|
||||
let mut restored = Vec::new();
|
||||
reader
|
||||
.stream
|
||||
.read_to_end(&mut restored)
|
||||
.await
|
||||
.expect("mid-size wiring reader should stream");
|
||||
|
||||
assert_eq!(restored, payload);
|
||||
assert!(
|
||||
crate::set_disk::test_get_object_reader_selected_mid_size(),
|
||||
"full get_object_reader path must select mid-size streaming (path id {})",
|
||||
crate::set_disk::test_get_object_reader_path_id()
|
||||
);
|
||||
assert!(
|
||||
crate::set_disk::coding::decode_reader::test_single_inflight_construction_count() > single_inflight_before,
|
||||
"mid-size get_object_reader wiring must construct SingleInFlight"
|
||||
);
|
||||
},
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn repeated_gets_reuse_the_set_erasure_shell() {
|
||||
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
|
||||
@@ -15493,6 +15930,161 @@ mod put_object_tmp_cleanup_tests {
|
||||
assert_eq!(body, b"new client body");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn conditional_put_does_not_block_reads_during_body_ingestion() {
|
||||
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
|
||||
let bucket = "conditional-put-nonblocking-read";
|
||||
let object = "object";
|
||||
for disk in &disk_stores {
|
||||
disk.make_volume(bucket).await.expect("bucket volume should be created");
|
||||
}
|
||||
|
||||
let mut initial_reader = PutObjReader::from_vec(b"old body".to_vec());
|
||||
let initial = set_disks
|
||||
.put_object(bucket, object, &mut initial_reader, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("initial object should be written");
|
||||
let initial_etag = initial.etag.clone().expect("initial object should have an etag");
|
||||
|
||||
let body = vec![b'c'; 64 * 1024];
|
||||
let split = body.len() / 2;
|
||||
let (mut source, stream) = tokio::io::duplex(64);
|
||||
let hash_reader = HashReader::from_stream(
|
||||
stream,
|
||||
i64::try_from(body.len()).expect("body length should fit i64"),
|
||||
i64::try_from(body.len()).expect("body length should fit i64"),
|
||||
None,
|
||||
None,
|
||||
false,
|
||||
)
|
||||
.expect("conditional hash reader should be created");
|
||||
let writer_store = Arc::clone(&set_disks);
|
||||
let etag_for_put = initial_etag.clone();
|
||||
let put = tokio::spawn(async move {
|
||||
let mut reader = PutObjReader::new(hash_reader);
|
||||
writer_store
|
||||
.put_object(
|
||||
bucket,
|
||||
object,
|
||||
&mut reader,
|
||||
&ObjectOptions {
|
||||
http_preconditions: Some(HTTPPreconditions {
|
||||
if_match: Some(etag_for_put),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
});
|
||||
|
||||
source
|
||||
.write_all(&body[..split])
|
||||
.await
|
||||
.expect("conditional PUT should consume the first half of the body");
|
||||
let info = tokio::time::timeout(
|
||||
Duration::from_secs(5),
|
||||
set_disks.get_object_info(bucket, object, &ObjectOptions::default()),
|
||||
)
|
||||
.await
|
||||
.expect("reads must not wait for the conditional PUT body")
|
||||
.expect("the old version must stay readable during body ingestion");
|
||||
assert_eq!(info.etag.as_deref(), Some(initial_etag.as_str()));
|
||||
|
||||
source
|
||||
.write_all(&body[split..])
|
||||
.await
|
||||
.expect("conditional PUT should consume the remaining body");
|
||||
drop(source);
|
||||
put.await
|
||||
.expect("conditional PUT task should join")
|
||||
.expect("conditional PUT should commit after the body completes");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn conditional_put_precondition_is_rechecked_at_commit() {
|
||||
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
|
||||
let bucket = "conditional-put-commit-recheck";
|
||||
let object = "object";
|
||||
for disk in &disk_stores {
|
||||
disk.make_volume(bucket).await.expect("bucket volume should be created");
|
||||
}
|
||||
|
||||
let mut initial_reader = PutObjReader::from_vec(b"old body".to_vec());
|
||||
let initial = set_disks
|
||||
.put_object(bucket, object, &mut initial_reader, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("initial object should be written");
|
||||
let initial_etag = initial.etag.clone().expect("initial object should have an etag");
|
||||
|
||||
let body = vec![b'c'; 64 * 1024];
|
||||
let split = body.len() / 2;
|
||||
let (mut source, stream) = tokio::io::duplex(64);
|
||||
let hash_reader = HashReader::from_stream(
|
||||
stream,
|
||||
i64::try_from(body.len()).expect("body length should fit i64"),
|
||||
i64::try_from(body.len()).expect("body length should fit i64"),
|
||||
None,
|
||||
None,
|
||||
false,
|
||||
)
|
||||
.expect("conditional hash reader should be created");
|
||||
let writer_store = Arc::clone(&set_disks);
|
||||
let put = tokio::spawn(async move {
|
||||
let mut reader = PutObjReader::new(hash_reader);
|
||||
writer_store
|
||||
.put_object(
|
||||
bucket,
|
||||
object,
|
||||
&mut reader,
|
||||
&ObjectOptions {
|
||||
http_preconditions: Some(HTTPPreconditions {
|
||||
if_match: Some(initial_etag),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
});
|
||||
|
||||
source
|
||||
.write_all(&body[..split])
|
||||
.await
|
||||
.expect("conditional PUT should consume the first half of the body");
|
||||
let mut interloper_reader = PutObjReader::from_vec(b"interloper body".to_vec());
|
||||
tokio::time::timeout(
|
||||
Duration::from_secs(5),
|
||||
set_disks.put_object(bucket, object, &mut interloper_reader, &ObjectOptions::default()),
|
||||
)
|
||||
.await
|
||||
.expect("the interloper write must not wait for the conditional PUT body")
|
||||
.expect("the interloper write should commit while the conditional PUT streams");
|
||||
|
||||
source
|
||||
.write_all(&body[split..])
|
||||
.await
|
||||
.expect("conditional PUT should consume the remaining body");
|
||||
drop(source);
|
||||
let err = put
|
||||
.await
|
||||
.expect("conditional PUT task should join")
|
||||
.expect_err("the conditional PUT must recheck its precondition under the commit lock");
|
||||
assert_eq!(err, StorageError::PreconditionFailed);
|
||||
|
||||
let mut reader = set_disks
|
||||
.get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
|
||||
.await
|
||||
.expect("the interloper object should remain readable");
|
||||
let mut body = Vec::new();
|
||||
reader
|
||||
.stream
|
||||
.read_to_end(&mut body)
|
||||
.await
|
||||
.expect("the interloper object should drain");
|
||||
assert_eq!(body, b"interloper body");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn metadata_copy_no_lock_aborts_after_outer_namespace_lock_loss() {
|
||||
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
|
||||
|
||||
@@ -52,6 +52,8 @@ use std::{
|
||||
};
|
||||
use tokio::io::{AsyncRead, AsyncWrite, ReadBuf};
|
||||
|
||||
pub(super) const GET_OBJECT_PATH_MID_SIZE_STREAMING: &str = "mid_size_streaming";
|
||||
|
||||
#[cfg(test)]
|
||||
use super::DEFAULT_GET_OBJECT_METADATA_CACHE_MAX_ENTRIES;
|
||||
#[cfg(test)]
|
||||
@@ -1340,6 +1342,80 @@ impl SetDisks {
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
#[hotpath::measure(impl_type = "SetDisks")]
|
||||
pub(super) async fn get_object_decode_reader_with_fileinfo(
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
erasure_cache: Arc<ErasureCache>,
|
||||
fi: &FileInfo,
|
||||
files: &[FileInfo],
|
||||
disks: &[Option<DiskStore>],
|
||||
set_index: usize,
|
||||
pool_index: usize,
|
||||
skip_verify_bitrot: bool,
|
||||
metrics_object_class: &'static str,
|
||||
metrics_size_bucket: &'static str,
|
||||
prefer_data_blocks_first_reader_setup: bool,
|
||||
) -> Result<GetCodecStreamingReaderBuildOutcome> {
|
||||
Self::get_object_decode_reader_with_fileinfo_inner(
|
||||
bucket,
|
||||
object,
|
||||
erasure_cache,
|
||||
fi,
|
||||
files,
|
||||
disks,
|
||||
set_index,
|
||||
pool_index,
|
||||
skip_verify_bitrot,
|
||||
metrics_object_class,
|
||||
metrics_size_bucket,
|
||||
prefer_data_blocks_first_reader_setup,
|
||||
get_codec_streaming_metrics_path(),
|
||||
false,
|
||||
false,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Build the bounded mid-size reader while allowing a degraded part to
|
||||
/// reuse the shard readers it just opened for an in-place legacy fallback.
|
||||
/// This avoids opening every shard twice when a healthy quorum requires
|
||||
/// reconstruction.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub(super) async fn get_object_mid_size_reader_with_fileinfo(
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
erasure_cache: Arc<ErasureCache>,
|
||||
fi: &FileInfo,
|
||||
files: &[FileInfo],
|
||||
disks: &[Option<DiskStore>],
|
||||
set_index: usize,
|
||||
pool_index: usize,
|
||||
skip_verify_bitrot: bool,
|
||||
metrics_object_class: &'static str,
|
||||
metrics_size_bucket: &'static str,
|
||||
prefer_data_blocks_first_reader_setup: bool,
|
||||
) -> Result<GetCodecStreamingReaderBuildOutcome> {
|
||||
Self::get_object_decode_reader_with_fileinfo_inner(
|
||||
bucket,
|
||||
object,
|
||||
erasure_cache,
|
||||
fi,
|
||||
files,
|
||||
disks,
|
||||
set_index,
|
||||
pool_index,
|
||||
skip_verify_bitrot,
|
||||
metrics_object_class,
|
||||
metrics_size_bucket,
|
||||
prefer_data_blocks_first_reader_setup,
|
||||
GET_OBJECT_PATH_MID_SIZE_STREAMING,
|
||||
true,
|
||||
true,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
async fn get_object_decode_reader_with_fileinfo_inner(
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
erasure_cache: Arc<ErasureCache>,
|
||||
@@ -1352,6 +1428,9 @@ impl SetDisks {
|
||||
metrics_object_class: &'static str,
|
||||
metrics_size_bucket: &'static str,
|
||||
prefer_data_blocks_first_reader_setup: bool,
|
||||
metrics_path: &'static str,
|
||||
allow_inplace_legacy_fallback: bool,
|
||||
single_inflight: bool,
|
||||
) -> Result<GetCodecStreamingReaderBuildOutcome> {
|
||||
let erasure = erasure_cache.get_for_file_info(fi)?;
|
||||
let (disks, files) = Self::shuffle_disks_and_parts_metadata_by_index(disks, files, fi);
|
||||
@@ -1375,10 +1454,12 @@ impl SetDisks {
|
||||
metrics_object_class,
|
||||
metrics_size_bucket,
|
||||
prefer_data_blocks_first_reader_setup,
|
||||
single_inflight,
|
||||
// Single-part objects keep the whole-request fallback: a degraded
|
||||
// sole part is detected before any byte streams, so the caller can
|
||||
// still hand the request to the legacy duplex path unchanged.
|
||||
false,
|
||||
allow_inplace_legacy_fallback,
|
||||
metrics_path,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
@@ -1427,10 +1508,12 @@ impl SetDisks {
|
||||
metrics_object_class,
|
||||
metrics_size_bucket,
|
||||
false,
|
||||
false,
|
||||
// The first part stays eager and keeps the whole-request fallback:
|
||||
// if part 1 is already degraded, the entire GET drops to the legacy
|
||||
// duplex path before a single byte is streamed (semantics unchanged).
|
||||
false,
|
||||
metrics_path,
|
||||
)
|
||||
.await?
|
||||
{
|
||||
@@ -1452,6 +1535,7 @@ impl SetDisks {
|
||||
skip_verify_bitrot,
|
||||
metrics_object_class,
|
||||
metrics_size_bucket,
|
||||
metrics_path,
|
||||
});
|
||||
let builder: LazyPartBuilder = Box::new(move |remaining_index| {
|
||||
let ctx = Arc::clone(&ctx);
|
||||
@@ -1472,18 +1556,20 @@ impl SetDisks {
|
||||
ctx.metrics_object_class,
|
||||
ctx.metrics_size_bucket,
|
||||
false,
|
||||
false,
|
||||
// backlog#879: later parts have already streamed earlier bytes,
|
||||
// so a whole-request fallback is impossible here. Degrade this
|
||||
// part in place to a legacy per-part decode reader instead of
|
||||
// failing the stream mid-flight.
|
||||
true,
|
||||
ctx.metrics_path,
|
||||
)
|
||||
.await
|
||||
})
|
||||
});
|
||||
|
||||
Ok(GetCodecStreamingReaderBuildOutcome::Reader(Box::new(
|
||||
LazyMultipartCodecStreamingReader::new(first_reader, total_parts, builder, get_codec_streaming_metrics_path()),
|
||||
LazyMultipartCodecStreamingReader::new(first_reader, total_parts, builder, metrics_path),
|
||||
)))
|
||||
}
|
||||
|
||||
@@ -1504,14 +1590,9 @@ impl SetDisks {
|
||||
metrics_object_class: &'static str,
|
||||
metrics_size_bucket: &'static str,
|
||||
prefer_data_blocks_first_reader_setup: bool,
|
||||
// backlog#879: when the codec streaming fast path cannot serve this part
|
||||
// (a shard is missing and reconstruction is required), `false` preserves
|
||||
// the historical whole-request fallback by returning `Fallback`, while
|
||||
// `true` degrades in place — building a legacy per-part decode reader that
|
||||
// reuses the shard readers already opened here. Only lazily-built later
|
||||
// parts pass `true`, so the eager first-part fallback semantics are
|
||||
// untouched and the common read path is never affected.
|
||||
single_inflight: bool,
|
||||
allow_inplace_legacy_fallback: bool,
|
||||
metrics_path: &'static str,
|
||||
) -> Result<GetCodecStreamingReaderBuildOutcome> {
|
||||
if part_length > part_size {
|
||||
return Err(Error::other("codec streaming reader part length exceeds part size"));
|
||||
@@ -1528,7 +1609,6 @@ impl SetDisks {
|
||||
let read_length = till_offset.saturating_sub(read_offset);
|
||||
|
||||
let stage_metrics_enabled = rustfs_io_metrics::get_stage_metrics_enabled();
|
||||
let metrics_path = get_codec_streaming_metrics_path();
|
||||
let reader_stage_metrics = stage_metrics_enabled.then_some(BitrotReaderStageMetrics {
|
||||
path: metrics_path,
|
||||
reader_construction_stage: GET_STAGE_READER_TASK_READER_CONSTRUCTION,
|
||||
@@ -1631,11 +1711,23 @@ impl SetDisks {
|
||||
.with_deferred_parity_handles(deferred_stripe_handles)
|
||||
.with_deferred_parity_reopeners(deferred_reopeners);
|
||||
let engine = build_get_codec_streaming_decode_engine(erasure.clone())?;
|
||||
let reader =
|
||||
coding::decode_reader::ErasureDecodeReader::new_with_metrics_path(source, engine, part_length, metrics_path)?;
|
||||
Ok(GetCodecStreamingReaderBuildOutcome::Reader(Box::new(
|
||||
coding::decode_reader::SyncErasureDecodeReader::new_with_metrics_path(reader, metrics_path),
|
||||
)))
|
||||
let reader = if single_inflight {
|
||||
coding::decode_reader::ErasureDecodeReader::new_single_inflight_with_metrics_path(
|
||||
source,
|
||||
engine,
|
||||
part_length,
|
||||
metrics_path,
|
||||
)?
|
||||
} else {
|
||||
coding::decode_reader::ErasureDecodeReader::new_with_metrics_path(source, engine, part_length, metrics_path)?
|
||||
};
|
||||
if single_inflight {
|
||||
Ok(GetCodecStreamingReaderBuildOutcome::Reader(Box::new(reader)))
|
||||
} else {
|
||||
Ok(GetCodecStreamingReaderBuildOutcome::Reader(Box::new(
|
||||
coding::decode_reader::SyncErasureDecodeReader::new_with_metrics_path(reader, metrics_path),
|
||||
)))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1751,6 +1843,7 @@ struct LazyCodecPartContext {
|
||||
skip_verify_bitrot: bool,
|
||||
metrics_object_class: &'static str,
|
||||
metrics_size_bucket: &'static str,
|
||||
metrics_path: &'static str,
|
||||
}
|
||||
|
||||
type LazyPartBuildHandle = tokio::task::JoinHandle<Result<GetCodecStreamingReaderBuildOutcome>>;
|
||||
@@ -4594,6 +4687,154 @@ mod tests {
|
||||
assert_eq!(body, part_data);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn mid_size_reader_restores_plain_objects_without_duplex() {
|
||||
for size in [256_usize * 1024, 1024_usize * 1024] {
|
||||
let payload = (0..size)
|
||||
.map(|index| u8::try_from(index % 251).expect("pattern byte fits u8"))
|
||||
.collect::<Vec<_>>();
|
||||
let erasure = coding::Erasure::new(4, 2, 1024 * 1024);
|
||||
let mut fi = codec_streaming_test_fileinfo(i64::try_from(size).expect("test size fits i64"), 1);
|
||||
fi.erasure.block_size = erasure.block_size;
|
||||
fi.erasure.distribution = (1..=erasure.total_shard_count()).collect();
|
||||
let files = codec_streaming_inline_files(&erasure, &payload).await;
|
||||
let (_dirs, disks) = local_test_disks(files.len(), CODEC_STREAMING_TEST_BUCKET).await;
|
||||
|
||||
let outcome = SetDisks::get_object_mid_size_reader_with_fileinfo(
|
||||
CODEC_STREAMING_TEST_BUCKET,
|
||||
CODEC_STREAMING_TEST_OBJECT,
|
||||
Arc::new(ErasureCache::new()),
|
||||
&fi,
|
||||
&files,
|
||||
&disks,
|
||||
0,
|
||||
0,
|
||||
false,
|
||||
"plain_single_part",
|
||||
"le_1mib",
|
||||
false,
|
||||
)
|
||||
.await
|
||||
.expect("mid-size reader setup should succeed");
|
||||
let GetCodecStreamingReaderBuildOutcome::Reader(mut reader) = outcome else {
|
||||
panic!("mid-size plain object should use streaming reader");
|
||||
};
|
||||
|
||||
let mut body = Vec::new();
|
||||
reader
|
||||
.read_to_end(&mut body)
|
||||
.await
|
||||
.expect("mid-size reader should restore full body");
|
||||
assert_eq!(body, payload, "mid-size reader must preserve every payload byte for object size {size}");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn mid_size_reader_restores_plain_objects_from_external_part_files() {
|
||||
for size in [256_usize * 1024, 1024_usize * 1024] {
|
||||
let payload = (0..size)
|
||||
.map(|index| u8::try_from(index % 251).expect("pattern byte fits u8"))
|
||||
.collect::<Vec<_>>();
|
||||
let erasure = coding::Erasure::new(4, 2, 1024 * 1024);
|
||||
let mut fi = codec_streaming_test_fileinfo(i64::try_from(size).expect("test size fits i64"), 1);
|
||||
fi.erasure.block_size = erasure.block_size;
|
||||
fi.erasure.distribution = (1..=erasure.total_shard_count()).collect();
|
||||
let (_dirs, disks) = local_test_disks(erasure.total_shard_count(), CODEC_STREAMING_TEST_BUCKET).await;
|
||||
let (data_dir, files) = codec_streaming_external_files(&erasure, &payload, &disks).await;
|
||||
fi.data_dir = Some(data_dir);
|
||||
|
||||
let outcome = SetDisks::get_object_mid_size_reader_with_fileinfo(
|
||||
CODEC_STREAMING_TEST_BUCKET,
|
||||
CODEC_STREAMING_TEST_OBJECT,
|
||||
Arc::new(ErasureCache::new()),
|
||||
&fi,
|
||||
&files,
|
||||
&disks,
|
||||
0,
|
||||
0,
|
||||
false,
|
||||
"plain_single_part",
|
||||
"le_1mib",
|
||||
false,
|
||||
)
|
||||
.await
|
||||
.expect("external mid-size reader setup should succeed");
|
||||
let GetCodecStreamingReaderBuildOutcome::Reader(mut reader) = outcome else {
|
||||
panic!("external plain object should use streaming reader");
|
||||
};
|
||||
|
||||
let mut body = Vec::new();
|
||||
reader
|
||||
.read_to_end(&mut body)
|
||||
.await
|
||||
.expect("external mid-size reader should restore full body");
|
||||
assert_eq!(
|
||||
body, payload,
|
||||
"external part files must preserve every payload byte for object size {size}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn minio_large_object_metadata_remains_reader_compatible() {
|
||||
let raw = rustfs_filemeta::test_data::create_minio_large_object_xlmeta().expect("load MinIO large-object fixture");
|
||||
let file_info = rustfs_filemeta::FileMeta::load(&raw)
|
||||
.expect("MinIO xl.meta should decode")
|
||||
.into_fileinfo("interop", "large.bin", "", false, false, true)
|
||||
.expect("MinIO xl.meta should materialize FileInfo");
|
||||
|
||||
file_info
|
||||
.validate_for_metadata_read()
|
||||
.expect("MinIO metadata should satisfy the reader validation contract");
|
||||
assert_eq!(file_info.size, 300_000);
|
||||
assert_eq!(file_info.parts.len(), 1);
|
||||
assert!(file_info.data_dir.is_some());
|
||||
assert!(!file_info.inline_data());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn mid_size_reader_reuses_open_readers_for_degraded_quorum() {
|
||||
let size = 256 * 1024;
|
||||
let payload = vec![0x5a; size];
|
||||
let erasure = coding::Erasure::new(4, 2, 1024 * 1024);
|
||||
let mut fi = codec_streaming_test_fileinfo(i64::try_from(size).expect("test size fits i64"), 1);
|
||||
fi.erasure.block_size = erasure.block_size;
|
||||
fi.erasure.distribution = (1..=erasure.total_shard_count()).collect();
|
||||
let mut files = codec_streaming_inline_files(&erasure, &payload).await;
|
||||
files[0].data = None;
|
||||
let (_dirs, disks) = local_test_disks(files.len(), CODEC_STREAMING_TEST_BUCKET).await;
|
||||
|
||||
let outcome = SetDisks::get_object_mid_size_reader_with_fileinfo(
|
||||
CODEC_STREAMING_TEST_BUCKET,
|
||||
CODEC_STREAMING_TEST_OBJECT,
|
||||
Arc::new(ErasureCache::new()),
|
||||
&fi,
|
||||
&files,
|
||||
&disks,
|
||||
0,
|
||||
0,
|
||||
false,
|
||||
"plain_single_part",
|
||||
"le_1mib",
|
||||
false,
|
||||
)
|
||||
.await
|
||||
.expect("degraded mid-size reader setup should retain read quorum");
|
||||
let GetCodecStreamingReaderBuildOutcome::Reader(mut reader) = outcome else {
|
||||
panic!("degraded quorum should use in-place legacy fallback reader");
|
||||
};
|
||||
|
||||
let mut body = Vec::new();
|
||||
reader
|
||||
.read_to_end(&mut body)
|
||||
.await
|
||||
.expect("degraded reader should reconstruct full body");
|
||||
assert_eq!(body, payload);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn get_object_with_fileinfo_restores_missing_inline_data_shard_and_submits_repair() {
|
||||
let part_data = b"abcdefgh";
|
||||
@@ -4654,6 +4895,8 @@ mod tests {
|
||||
"test-size-bucket",
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
get_codec_streaming_metrics_path(),
|
||||
)
|
||||
.await;
|
||||
assert!(oversized.is_err(), "part_length > part_size must be rejected");
|
||||
@@ -4674,6 +4917,8 @@ mod tests {
|
||||
"test-size-bucket",
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
get_codec_streaming_metrics_path(),
|
||||
)
|
||||
.await;
|
||||
assert!(missing_quorum.is_err(), "reader setup must fail closed when no shard can answer");
|
||||
@@ -5208,7 +5453,7 @@ mod tests {
|
||||
Ok(decoded)
|
||||
}
|
||||
|
||||
async fn codec_streaming_inline_files(erasure: &coding::Erasure, part_data: &'static [u8]) -> Vec<FileInfo> {
|
||||
async fn codec_streaming_inline_files(erasure: &coding::Erasure, part_data: &[u8]) -> Vec<FileInfo> {
|
||||
let shards = erasure.encode_data(part_data).expect("test part should encode");
|
||||
let distribution = (1..=erasure.total_shard_count()).collect::<Vec<_>>();
|
||||
let mut files = Vec::with_capacity(shards.len());
|
||||
@@ -5231,6 +5476,27 @@ mod tests {
|
||||
files
|
||||
}
|
||||
|
||||
async fn codec_streaming_external_files(
|
||||
erasure: &coding::Erasure,
|
||||
part_data: &[u8],
|
||||
disks: &[Option<crate::disk::DiskStore>],
|
||||
) -> (Uuid, Vec<FileInfo>) {
|
||||
let data_dir = Uuid::from_u128(0x2060_0000_0000_0000_0000_0000_0000_0001);
|
||||
let mut files = codec_streaming_inline_files(erasure, part_data).await;
|
||||
for (index, file) in files.iter_mut().enumerate() {
|
||||
let shard = file.data.take().expect("external fixture shard should be encoded");
|
||||
file.data_dir = Some(data_dir);
|
||||
let path = format!("{CODEC_STREAMING_TEST_OBJECT}/{data_dir}/part.1");
|
||||
disks[index]
|
||||
.as_ref()
|
||||
.expect("external fixture disk should be online")
|
||||
.write_all(CODEC_STREAMING_TEST_BUCKET, &path, shard)
|
||||
.await
|
||||
.expect("external fixture shard should be written");
|
||||
}
|
||||
(data_dir, files)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn bitrot_reader_setup_stops_at_read_quorum() {
|
||||
let setup = setup_inline_bitrot_readers(
|
||||
|
||||
@@ -99,6 +99,8 @@ fn preflight_startup_rpc_secret_with(
|
||||
const LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES: usize = 6;
|
||||
const LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY: Duration = Duration::from_secs(60 * 3);
|
||||
const LOCAL_DECOMMISSION_RESUME_RETRY_DELAY: Duration = Duration::from_secs(30);
|
||||
const REBALANCE_INITIAL_RESUME_DELAY: Duration = Duration::from_secs(10);
|
||||
const REBALANCE_RESUME_RETRY_DELAY: Duration = Duration::from_secs(10);
|
||||
|
||||
fn should_retry_local_decommission_resume(err: &Error, attempt: usize) -> bool {
|
||||
matches!(err, Error::ConfigNotFound) && attempt < LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES
|
||||
@@ -108,8 +110,12 @@ fn should_retry_format_load(err: &Error) -> bool {
|
||||
!matches!(err, Error::CorruptedFormat)
|
||||
}
|
||||
|
||||
fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_meta_loaded: bool) -> bool {
|
||||
rebalance_meta_loaded && !decommission_running
|
||||
fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_resume_required: bool) -> bool {
|
||||
rebalance_resume_required && !decommission_running
|
||||
}
|
||||
|
||||
fn should_defer_rebalance_auto_start(distributed: bool, fleet_proof_available: bool) -> bool {
|
||||
distributed && !fleet_proof_available
|
||||
}
|
||||
|
||||
fn should_schedule_local_decommission_resume(
|
||||
@@ -127,6 +133,17 @@ async fn wait_for_local_decommission_resume_delay(rx: &CancellationToken, delay:
|
||||
}
|
||||
}
|
||||
|
||||
async fn wait_for_rebalance_resume_delay(rx: &CancellationToken, delay: Duration) -> bool {
|
||||
tokio::select! {
|
||||
_ = rx.cancelled() => false,
|
||||
_ = tokio::time::sleep(delay) => true,
|
||||
}
|
||||
}
|
||||
|
||||
async fn wait_for_rebalance_resume_retry(rx: &CancellationToken) -> bool {
|
||||
wait_for_rebalance_resume_delay(rx, REBALANCE_RESUME_RETRY_DELAY).await
|
||||
}
|
||||
|
||||
fn resolve_store_init_stage_result(result: Result<()>, stage: &str) -> Result<()> {
|
||||
result.map_err(|err| Error::other(format!("store init failed during {stage}: {err}")))
|
||||
}
|
||||
@@ -283,6 +300,71 @@ async fn resume_local_decommission_after_init(store: Arc<ECStore>, rx: Cancellat
|
||||
}
|
||||
}
|
||||
|
||||
async fn resume_rebalance_after_init(store: Arc<ECStore>, rx: CancellationToken) {
|
||||
if !wait_for_rebalance_resume_delay(&rx, REBALANCE_INITIAL_RESUME_DELAY).await {
|
||||
return;
|
||||
}
|
||||
|
||||
loop {
|
||||
if rx.is_cancelled() {
|
||||
return;
|
||||
}
|
||||
|
||||
let resume_required = store
|
||||
.rebalance_meta
|
||||
.read()
|
||||
.await
|
||||
.as_ref()
|
||||
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation);
|
||||
if !resume_required {
|
||||
return;
|
||||
}
|
||||
|
||||
if should_defer_rebalance_auto_start(
|
||||
store.ctx.is_dist_erasure().await,
|
||||
crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_some(),
|
||||
) {
|
||||
if !wait_for_rebalance_resume_retry(&rx).await {
|
||||
return;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
match store.start_rebalance().await {
|
||||
Ok(()) => return,
|
||||
Err(err) if crate::core::pools::is_pool_activation_fleet_proof_error(&err) => {
|
||||
warn!(
|
||||
event = EVENT_ECSTORE_INIT_STATUS,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_STORE_INIT,
|
||||
stage = "start_rebalance",
|
||||
state = "retrying",
|
||||
reason = "fleet_capability_proof_unavailable",
|
||||
error = %err,
|
||||
retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
|
||||
"Retrying deferred rebalance auto-start"
|
||||
);
|
||||
if !wait_for_rebalance_resume_retry(&rx).await {
|
||||
return;
|
||||
}
|
||||
}
|
||||
Err(err) => {
|
||||
error!(
|
||||
event = EVENT_ECSTORE_INIT_STATUS,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_STORE_INIT,
|
||||
stage = "start_rebalance",
|
||||
state = "failed",
|
||||
reason = "deferred_resume_failed",
|
||||
error = %err,
|
||||
"Failed to resume rebalance after store initialization"
|
||||
);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl ECStore {
|
||||
/// Validate topology and process storage-class overrides before any disk is opened.
|
||||
pub fn validate_startup_storage_class(endpoint_pools: &EndpointServerPools) -> Result<()> {
|
||||
@@ -574,12 +656,49 @@ impl ECStore {
|
||||
}
|
||||
|
||||
resolve_store_init_stage_result(self.load_rebalance_meta().await, "load_rebalance_meta")?;
|
||||
let rebalance_meta_loaded = self.rebalance_meta.read().await.is_some();
|
||||
let rebalance_resume_required = {
|
||||
let rebalance_meta = self.rebalance_meta.read().await;
|
||||
rebalance_meta
|
||||
.as_ref()
|
||||
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
|
||||
};
|
||||
let decommission_running =
|
||||
pool_meta_has_active_decommission(&installed_pool_meta) || self.is_decommission_running().await;
|
||||
if should_auto_start_rebalance_after_init(decommission_running, rebalance_meta_loaded) {
|
||||
resolve_store_init_stage_result(self.start_rebalance().await, "start_rebalance")?;
|
||||
} else if decommission_running && rebalance_meta_loaded {
|
||||
let distributed = self.ctx.is_dist_erasure().await;
|
||||
let fleet_proof_available = crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof().is_some();
|
||||
let mut rebalance_auto_start_deferred = false;
|
||||
if should_auto_start_rebalance_after_init(decommission_running, rebalance_resume_required) {
|
||||
if should_defer_rebalance_auto_start(distributed, fleet_proof_available) {
|
||||
rebalance_auto_start_deferred = true;
|
||||
warn!(
|
||||
event = EVENT_ECSTORE_INIT_STATUS,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_STORE_INIT,
|
||||
stage = "start_rebalance",
|
||||
state = "deferred",
|
||||
reason = "fleet_capability_proof_unavailable",
|
||||
retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
|
||||
"Deferred rebalance auto-start until a live fleet capability proof is available"
|
||||
);
|
||||
} else if let Err(err) = self.start_rebalance().await {
|
||||
if crate::core::pools::is_pool_activation_fleet_proof_error(&err) {
|
||||
rebalance_auto_start_deferred = true;
|
||||
warn!(
|
||||
event = EVENT_ECSTORE_INIT_STATUS,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_STORE_INIT,
|
||||
stage = "start_rebalance",
|
||||
state = "deferred",
|
||||
reason = "fleet_capability_proof_changed",
|
||||
error = %err,
|
||||
retry_delay_secs = REBALANCE_RESUME_RETRY_DELAY.as_secs(),
|
||||
"Deferred rebalance auto-start after the fleet capability proof changed"
|
||||
);
|
||||
} else {
|
||||
return resolve_store_init_stage_result(Err(err), "start_rebalance");
|
||||
}
|
||||
}
|
||||
} else if decommission_running && rebalance_resume_required {
|
||||
warn!(
|
||||
event = EVENT_ECSTORE_INIT_STATUS,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
@@ -616,12 +735,13 @@ impl ECStore {
|
||||
.is_ok();
|
||||
if should_schedule_local_decommission_resume(&local_pool_indices, pool_meta_replica_state, pool_meta_write_safe) {
|
||||
let store = self.clone();
|
||||
let decommission_rx = rx.clone();
|
||||
|
||||
tokio::spawn(async move {
|
||||
if !wait_for_local_decommission_resume_delay(&rx, LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY).await {
|
||||
if !wait_for_local_decommission_resume_delay(&decommission_rx, LOCAL_DECOMMISSION_INITIAL_RESUME_DELAY).await {
|
||||
return;
|
||||
}
|
||||
resume_local_decommission_after_init(store, rx, local_pool_indices).await;
|
||||
resume_local_decommission_after_init(store, decommission_rx, local_pool_indices).await;
|
||||
});
|
||||
} else if !local_pool_indices.is_empty() {
|
||||
error!(
|
||||
@@ -648,6 +768,11 @@ impl ECStore {
|
||||
info!("TierConfigMgr init error: {}", err);
|
||||
}
|
||||
|
||||
if rebalance_auto_start_deferred {
|
||||
let store = self.clone();
|
||||
tokio::spawn(resume_rebalance_after_init(store, rx));
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -665,7 +790,8 @@ mod tests {
|
||||
load_pool_meta_for_startup, persist_pool_meta_for_startup_if_safe, pool_first_endpoint_is_local,
|
||||
pool_meta_has_active_decommission, preflight_startup_rpc_secret_with, resolve_startup_pool_defaults_with,
|
||||
resolve_store_init_stage_result, save_validated_pool_meta_for_startup, should_auto_start_rebalance_after_init,
|
||||
should_retry_format_load, should_retry_local_decommission_resume, wait_for_local_decommission_resume_delay,
|
||||
should_defer_rebalance_auto_start, should_retry_format_load, should_retry_local_decommission_resume,
|
||||
wait_for_local_decommission_resume_delay,
|
||||
};
|
||||
#[cfg(feature = "test-util")]
|
||||
use crate::disk::DiskAPI;
|
||||
@@ -1450,7 +1576,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_should_auto_start_rebalance_after_init_allows_loaded_rebalance_without_decommission() {
|
||||
fn test_should_auto_start_rebalance_after_init_allows_active_rebalance_without_decommission() {
|
||||
assert!(should_auto_start_rebalance_after_init(false, true));
|
||||
}
|
||||
|
||||
@@ -1460,10 +1586,17 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_should_auto_start_rebalance_after_init_rejects_missing_rebalance_meta() {
|
||||
fn test_should_auto_start_rebalance_after_init_rejects_terminal_or_missing_rebalance() {
|
||||
assert!(!should_auto_start_rebalance_after_init(false, false));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_should_defer_rebalance_auto_start_only_without_distributed_fleet_proof() {
|
||||
assert!(should_defer_rebalance_auto_start(true, false));
|
||||
assert!(!should_defer_rebalance_auto_start(true, true));
|
||||
assert!(!should_defer_rebalance_auto_start(false, false));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_store_init_recovery_skips_rebalance_when_decommission_metadata_is_active() {
|
||||
let pool_meta = init_test_pool_meta(Some(PoolDecommissionInfo {
|
||||
@@ -1473,22 +1606,69 @@ mod tests {
|
||||
canceled: false,
|
||||
..Default::default()
|
||||
}));
|
||||
let rebalance_meta = Some(RebalanceMeta::default());
|
||||
let rebalance_meta = Some(RebalanceMeta {
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Started,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
});
|
||||
|
||||
assert!(!should_auto_start_rebalance_after_init(
|
||||
pool_meta_has_active_decommission(&pool_meta),
|
||||
rebalance_meta.is_some()
|
||||
rebalance_meta
|
||||
.as_ref()
|
||||
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_store_init_recovery_allows_rebalance_when_only_rebalance_metadata_exists() {
|
||||
fn test_store_init_recovery_allows_active_rebalance_without_decommission() {
|
||||
let pool_meta = init_test_pool_meta(None);
|
||||
let rebalance_meta = Some(RebalanceMeta::default());
|
||||
let rebalance_meta = Some(RebalanceMeta {
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Started,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
});
|
||||
|
||||
assert!(should_auto_start_rebalance_after_init(
|
||||
pool_meta_has_active_decommission(&pool_meta),
|
||||
rebalance_meta.is_some()
|
||||
rebalance_meta
|
||||
.as_ref()
|
||||
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_store_init_recovery_skips_completed_rebalance_metadata() {
|
||||
let pool_meta = init_test_pool_meta(None);
|
||||
let rebalance_meta = Some(RebalanceMeta {
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Completed,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
});
|
||||
|
||||
assert!(!should_auto_start_rebalance_after_init(
|
||||
pool_meta_has_active_decommission(&pool_meta),
|
||||
rebalance_meta
|
||||
.as_ref()
|
||||
.is_some_and(crate::services::rebalance::rebalance_requires_worker_activation)
|
||||
));
|
||||
}
|
||||
|
||||
|
||||
@@ -2110,9 +2110,18 @@ fn build_list_versions_next_marker(
|
||||
cache_id: Option<&str>,
|
||||
) -> (Option<String>, Option<String>) {
|
||||
if let Some(last) = objects.last() {
|
||||
// A null version carries the synthesized `Some(Uuid::nil())` identity
|
||||
// here; advertise it as the literal `null` marker so a resumed listing
|
||||
// parses it back to `VersionMarker::Null` instead of a nil UUID that
|
||||
// `find_version_index` can never match (issue #6745).
|
||||
(
|
||||
Some(append_list_cache_id_to_marker(last.name.clone(), cache_id)),
|
||||
Some(last.version_id.map(|v| v.to_string()).unwrap_or_else(|| "null".to_string())),
|
||||
Some(
|
||||
last.version_id
|
||||
.filter(|v| !v.is_nil())
|
||||
.map(|v| v.to_string())
|
||||
.unwrap_or_else(|| "null".to_string()),
|
||||
),
|
||||
)
|
||||
} else if let Some(last_prefix) = prefixes.last() {
|
||||
(Some(append_list_cache_id_to_marker(last_prefix.clone(), cache_id)), None)
|
||||
@@ -7759,6 +7768,31 @@ mod test {
|
||||
out
|
||||
}
|
||||
|
||||
// A truncated versions listing ending on a null version must advertise the
|
||||
// literal `null` continuation marker: a nil UUID parses to
|
||||
// `VersionMarker::Version(nil)`, which no stored version matches, so the
|
||||
// resumed page would replay every version (issue #6745).
|
||||
#[test]
|
||||
fn build_list_versions_next_marker_reports_null_for_nil_version_id() {
|
||||
let null_version = ObjectInfo {
|
||||
name: "obj-a".to_owned(),
|
||||
version_id: Some(uuid::Uuid::nil()),
|
||||
..Default::default()
|
||||
};
|
||||
let real_version = ObjectInfo {
|
||||
name: "obj-b".to_owned(),
|
||||
version_id: Some(uuid::Uuid::from_u128(7)),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let (next_marker, next_version_idmarker) = super::build_list_versions_next_marker(&[null_version], &[], None);
|
||||
assert_eq!(next_marker.as_deref(), Some("obj-a"));
|
||||
assert_eq!(next_version_idmarker.as_deref(), Some("null"));
|
||||
|
||||
let (_, next_version_idmarker) = super::build_list_versions_next_marker(&[real_version], &[], None);
|
||||
assert_eq!(next_version_idmarker.as_deref(), Some(uuid::Uuid::from_u128(7).to_string().as_str()));
|
||||
}
|
||||
|
||||
// ECA-03 / #944: a page whose raw keys fully collapse into fewer than max_keys
|
||||
// common prefixes must still report truncation and carry a continuation marker,
|
||||
// otherwise every key beyond the scan window is silently dropped.
|
||||
|
||||
@@ -16,6 +16,8 @@ use thiserror::Error;
|
||||
|
||||
use super::heal::{DiskError, EcstoreError};
|
||||
|
||||
const HEAL_DANGLING_DELETE_GRACE_MESSAGE: &str = "dangling object deletion deferred by heal grace window";
|
||||
|
||||
/// Custom error type for heal operations
|
||||
/// This enum defines various error variants that can occur during
|
||||
/// the execution of heal-related tasks, such as I/O errors, storage errors,
|
||||
@@ -98,6 +100,9 @@ impl Error {
|
||||
// them.
|
||||
Error::Storage(EcstoreError::Lock(lock_err)) => !lock_err.is_fatal(),
|
||||
Error::Storage(err) => {
|
||||
if err.is_dangling_delete_grace() {
|
||||
return true;
|
||||
}
|
||||
err.is_quorum_error()
|
||||
|| matches!(
|
||||
err,
|
||||
@@ -110,6 +115,9 @@ impl Error {
|
||||
|| is_recoverable_heal_error_message(&err.to_string())
|
||||
}
|
||||
Error::Disk(err) => {
|
||||
if err.is_dangling_delete_grace() {
|
||||
return true;
|
||||
}
|
||||
matches!(
|
||||
err,
|
||||
DiskError::DiskNotFound
|
||||
@@ -127,6 +135,18 @@ impl Error {
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn is_dangling_delete_grace(&self) -> bool {
|
||||
match self {
|
||||
Error::Storage(err) => err.is_dangling_delete_grace(),
|
||||
Error::Disk(err) => err.is_dangling_delete_grace(),
|
||||
Error::Io(err) => DiskError::io_error_is_dangling_delete_grace(err),
|
||||
Error::TaskExecutionFailed { message } | Error::Other(message) => {
|
||||
message.contains(HEAL_DANGLING_DELETE_GRACE_MESSAGE)
|
||||
}
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Documented substring fallback for errors that reach heal with their typed
|
||||
|
||||
@@ -159,9 +159,13 @@ impl ErasureSetHealer {
|
||||
|
||||
/// Classify an error returned by [`HealStorageAPI::heal_object`].
|
||||
///
|
||||
/// Both the inner `Ok((_, Some(err)))` and the outer `Err(err)` produced by
|
||||
/// `heal_object` wrap `Error::Storage(StorageError)`, so match on that.
|
||||
/// Most heal object failures wrap `Error::Storage(StorageError)`, while
|
||||
/// compatibility markers can also arrive through Disk/Io/task wrappers.
|
||||
fn classify_heal_object_error(err: &Error) -> HealObjectOutcome {
|
||||
if err.is_dangling_delete_grace() {
|
||||
return HealObjectOutcome::Transient;
|
||||
}
|
||||
|
||||
let Error::Storage(se) = err else {
|
||||
return HealObjectOutcome::Failed;
|
||||
};
|
||||
@@ -1459,6 +1463,7 @@ mod tests {
|
||||
// genuine object absence, or transient failures get recorded as "healed" and
|
||||
// permanently skipped.
|
||||
use super::{EcstoreError, Error, HealObjectOutcome};
|
||||
use crate::heal::DiskError;
|
||||
|
||||
fn classify(err: EcstoreError) -> HealObjectOutcome {
|
||||
ErasureSetHealer::classify_heal_object_error(&Error::Storage(err))
|
||||
@@ -1479,6 +1484,16 @@ mod tests {
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dangling_delete_grace_is_transient() {
|
||||
assert!(matches!(
|
||||
ErasureSetHealer::classify_heal_object_error(&Error::Disk(DiskError::other(
|
||||
"dangling object deletion deferred by heal grace window; retry_after_secs=3599; grace_secs=3600"
|
||||
))),
|
||||
HealObjectOutcome::Transient
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn genuine_object_absence_is_absent() {
|
||||
assert!(matches!(classify(EcstoreError::FileNotFound), HealObjectOutcome::Absent));
|
||||
|
||||
@@ -20,7 +20,10 @@ use crate::heal::{
|
||||
};
|
||||
use crate::{Error, Result};
|
||||
use metrics::{counter, gauge};
|
||||
use rustfs_concurrency::{AdmissionState, WorkloadAdmissionSnapshotProvider, WorkloadClass};
|
||||
use rustfs_concurrency::WorkloadAdmissionSnapshotProvider;
|
||||
use rustfs_concurrency::workload::{ForegroundPressure, foreground_pressure};
|
||||
#[cfg(test)]
|
||||
use rustfs_concurrency::{AdmissionState, WorkloadClass};
|
||||
use rustfs_heal_contracts::heal_channel::{
|
||||
HealAdmissionDropReason, HealAdmissionReceipt, HealAdmissionResult, HealRequestSource,
|
||||
};
|
||||
@@ -813,40 +816,11 @@ impl HealManager {
|
||||
}
|
||||
|
||||
let provider = provider.as_ref()?;
|
||||
let snapshot = provider.workload_admission_snapshot();
|
||||
[
|
||||
(WorkloadClass::ForegroundRead, config.mainline_read_utilization_high_percent),
|
||||
(WorkloadClass::ForegroundWrite, config.mainline_write_utilization_high_percent),
|
||||
]
|
||||
.into_iter()
|
||||
.filter_map(|(class, threshold_pct)| {
|
||||
if threshold_pct == 0 {
|
||||
return None;
|
||||
}
|
||||
|
||||
let entry = snapshot.get(class)?;
|
||||
let usage_pct = if matches!(entry.state, AdmissionState::Saturated) {
|
||||
100
|
||||
} else {
|
||||
let limit = entry.limit?;
|
||||
if limit == 0 {
|
||||
return None;
|
||||
}
|
||||
entry
|
||||
.active
|
||||
.unwrap_or(0)
|
||||
.saturating_mul(100)
|
||||
.checked_div(limit)
|
||||
.unwrap_or(100)
|
||||
};
|
||||
|
||||
(usage_pct >= threshold_pct).then_some(ForegroundPressure {
|
||||
class,
|
||||
usage_pct,
|
||||
threshold_pct,
|
||||
})
|
||||
})
|
||||
.max_by_key(|pressure| pressure.usage_pct)
|
||||
foreground_pressure(
|
||||
&provider.workload_admission_snapshot(),
|
||||
config.mainline_read_utilization_high_percent,
|
||||
config.mainline_write_utilization_high_percent,
|
||||
)
|
||||
}
|
||||
|
||||
fn schedule_mainline_throttle_recheck(notify: Arc<Notify>, delay: Duration) {
|
||||
|
||||
@@ -78,23 +78,6 @@ pub(super) enum QueuePushOutcome {
|
||||
Merged,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub(super) struct ForegroundPressure {
|
||||
pub(super) class: WorkloadClass,
|
||||
pub(super) usage_pct: usize,
|
||||
pub(super) threshold_pct: usize,
|
||||
}
|
||||
|
||||
impl ForegroundPressure {
|
||||
pub(super) const fn reason(self) -> &'static str {
|
||||
match self.class {
|
||||
WorkloadClass::ForegroundRead => "foreground_read_pressure",
|
||||
WorkloadClass::ForegroundWrite => "foreground_write_pressure",
|
||||
_ => "foreground_pressure",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub(super) struct CompletedHealStatus {
|
||||
pub(super) heal_type: HealType,
|
||||
|
||||
@@ -682,6 +682,10 @@ impl HealTask {
|
||||
Self::is_data_usage_cache_object(bucket, object) && Self::is_transient_lock_or_timeout_error(err)
|
||||
}
|
||||
|
||||
fn is_dangling_delete_grace_error(err: &Error) -> bool {
|
||||
err.is_dangling_delete_grace()
|
||||
}
|
||||
|
||||
fn is_no_heal_required_error(err: &Error) -> bool {
|
||||
match err {
|
||||
Error::Storage(EcstoreError::NoHealRequired) | Error::Disk(DiskError::NoHealRequired) => true,
|
||||
@@ -746,6 +750,30 @@ impl HealTask {
|
||||
true
|
||||
}
|
||||
|
||||
async fn skip_dangling_delete_grace_error(&self, bucket: &str, object: &str, err: &Error) -> bool {
|
||||
if !Self::is_dangling_delete_grace_error(err) {
|
||||
return false;
|
||||
}
|
||||
|
||||
warn!(
|
||||
target: "rustfs::heal::task",
|
||||
event = EVENT_HEAL_OBJECT_RESULT,
|
||||
component = LOG_COMPONENT_HEAL,
|
||||
subsystem = LOG_SUBSYSTEM_OBJECT,
|
||||
task_id = %self.id,
|
||||
bucket,
|
||||
object,
|
||||
result = "dangling_delete_grace_skip",
|
||||
error = %err,
|
||||
"Heal object dangling cleanup deferred by grace window"
|
||||
);
|
||||
let mut progress = self.progress.write().await;
|
||||
progress.set_current_object(Some(format!("skipped: {bucket}/{object}")));
|
||||
progress.update_object_progress(1, 0, 0, 1, 0);
|
||||
progress.update_stage(3, 3);
|
||||
true
|
||||
}
|
||||
|
||||
async fn skip_scanner_synthetic_object_dir_missing(&self, bucket: &str, object: &str, err: &Error) -> bool {
|
||||
if self.source != HealRequestSource::Scanner || !is_missing_object_dir_heal_result(object, err) {
|
||||
return false;
|
||||
|
||||
@@ -359,7 +359,21 @@ impl HealTask {
|
||||
};
|
||||
|
||||
if let Some(err) = error {
|
||||
if Self::should_skip_data_usage_cache_heal_error(bucket, object, &err) {
|
||||
if Self::is_dangling_delete_grace_error(&err) {
|
||||
telemetry_unknown |= !increment_counter(&mut skipped);
|
||||
warn!(
|
||||
target: "rustfs::heal::task",
|
||||
event = EVENT_HEAL_BUCKET_RESULT,
|
||||
component = LOG_COMPONENT_HEAL,
|
||||
subsystem = LOG_SUBSYSTEM_TASK,
|
||||
task_id = %self.id,
|
||||
bucket,
|
||||
object,
|
||||
result = "dangling_delete_grace_skip",
|
||||
error = %err,
|
||||
"Heal bucket object dangling cleanup deferred by grace window"
|
||||
);
|
||||
} else if Self::should_skip_data_usage_cache_heal_error(bucket, object, &err) {
|
||||
telemetry_unknown |= !increment_counter(&mut skipped);
|
||||
warn!(
|
||||
target: "rustfs::heal::task",
|
||||
|
||||
@@ -169,6 +169,10 @@ impl HealTask {
|
||||
match heal_result {
|
||||
Ok((result, error)) => {
|
||||
if let Some(e) = error {
|
||||
if self.skip_dangling_delete_grace_error(bucket, object, &e).await {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
if self.skip_data_usage_cache_heal_error(bucket, object, &e).await {
|
||||
return Ok(());
|
||||
}
|
||||
@@ -257,6 +261,10 @@ impl HealTask {
|
||||
Err(Error::TaskCancelled) => Err(Error::TaskCancelled),
|
||||
Err(Error::TaskTimeout) => Err(Error::TaskTimeout),
|
||||
Err(e) => {
|
||||
if self.skip_dangling_delete_grace_error(bucket, object, &e).await {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
if self.skip_data_usage_cache_heal_error(bucket, object, &e).await {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
@@ -705,6 +705,7 @@ fn replacement_identity(
|
||||
enum MockHealObjectOutcome {
|
||||
OkWithOtherError(&'static str),
|
||||
ErrOther(&'static str),
|
||||
DanglingGraceDeferred,
|
||||
RetryableReadQuorum,
|
||||
RetryableSlowDown,
|
||||
PermanentOther(&'static str),
|
||||
@@ -806,6 +807,12 @@ impl HealStorageAPI for MockStorage {
|
||||
.and_then(VecDeque::pop_front)
|
||||
{
|
||||
return match outcome {
|
||||
MockHealObjectOutcome::DanglingGraceDeferred => Ok((
|
||||
HealResultItem::default(),
|
||||
Some(Error::Disk(DiskError::other(
|
||||
"dangling object deletion deferred by heal grace window; retry_after_secs=3599; grace_secs=3600",
|
||||
))),
|
||||
)),
|
||||
MockHealObjectOutcome::RetryableReadQuorum => Err(Error::Storage(EcstoreError::InsufficientReadQuorum(
|
||||
bucket.to_string(),
|
||||
object.to_string(),
|
||||
@@ -820,6 +827,12 @@ impl HealStorageAPI for MockStorage {
|
||||
}
|
||||
if let Some(outcome) = self.heal_object_outcome.lock().unwrap().take() {
|
||||
return match outcome {
|
||||
MockHealObjectOutcome::DanglingGraceDeferred => Ok((
|
||||
HealResultItem::default(),
|
||||
Some(Error::Disk(DiskError::other(
|
||||
"dangling object deletion deferred by heal grace window; retry_after_secs=3599; grace_secs=3600",
|
||||
))),
|
||||
)),
|
||||
MockHealObjectOutcome::OkWithOtherError(message) => Ok((HealResultItem::default(), Some(Error::other(message)))),
|
||||
MockHealObjectOutcome::ErrOther(message) | MockHealObjectOutcome::PermanentOther(message) => {
|
||||
Err(Error::other(message))
|
||||
@@ -1310,6 +1323,31 @@ async fn test_cluster_heal_visits_bucket_objects() {
|
||||
assert!(matches!(task.get_status().await, HealTaskStatus::Completed));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn object_heal_skips_dangling_delete_grace_without_failing_task() {
|
||||
let storage = Arc::new(MockStorage {
|
||||
heal_object_outcome: Mutex::new(Some(MockHealObjectOutcome::DanglingGraceDeferred)),
|
||||
..Default::default()
|
||||
});
|
||||
let task = HealTask::from_request(
|
||||
HealRequest::object("bucket-a".to_string(), "recent.txt".to_string(), None),
|
||||
storage.clone(),
|
||||
);
|
||||
|
||||
task.execute()
|
||||
.await
|
||||
.expect("grace-protected dangling cleanup should be reported as a skipped object");
|
||||
|
||||
assert!(matches!(task.get_status().await, HealTaskStatus::Completed));
|
||||
assert!(storage.healed_objects.lock().unwrap().is_empty());
|
||||
let progress = task.get_progress().await;
|
||||
assert_eq!(progress.current_object.as_deref(), Some("skipped: bucket-a/recent.txt"));
|
||||
assert_eq!(progress.objects_scanned, 1);
|
||||
assert_eq!(progress.objects_healed, 0);
|
||||
assert_eq!(progress.objects_failed, 0);
|
||||
assert_eq!(progress.skipped_objects, 1);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn test_recursive_bucket_heal_retries_only_retryable_objects() {
|
||||
let storage = Arc::new(MockStorage::default());
|
||||
@@ -1345,6 +1383,43 @@ async fn test_recursive_bucket_heal_retries_only_retryable_objects() {
|
||||
assert_eq!(progress.objects_failed, 0);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn recursive_bucket_heal_skips_dangling_delete_grace_without_batch_failure() {
|
||||
let storage = Arc::new(MockStorage::default());
|
||||
storage
|
||||
.heal_object_outcomes
|
||||
.lock()
|
||||
.unwrap()
|
||||
.insert("object-a".to_string(), VecDeque::from([MockHealObjectOutcome::DanglingGraceDeferred]));
|
||||
let request = HealRequest::new(
|
||||
HealType::Bucket {
|
||||
bucket: "bucket-a".to_string(),
|
||||
},
|
||||
HealOptions {
|
||||
recursive: true,
|
||||
timeout: None,
|
||||
..Default::default()
|
||||
},
|
||||
HealPriority::Normal,
|
||||
);
|
||||
let task = HealTask::from_request(request, storage.clone());
|
||||
|
||||
task.heal_bucket("bucket-a")
|
||||
.await
|
||||
.expect("grace-protected dangling cleanup should not fail the bucket heal batch");
|
||||
|
||||
assert_eq!(
|
||||
storage.heal_object_calls.lock().unwrap().as_slice(),
|
||||
["object-a".to_string(), "object-b".to_string()]
|
||||
);
|
||||
assert_eq!(storage.healed_objects.lock().unwrap().as_slice(), ["object-b".to_string()]);
|
||||
let progress = task.get_progress().await;
|
||||
assert_eq!(progress.objects_scanned, 2);
|
||||
assert_eq!(progress.objects_healed, 1);
|
||||
assert_eq!(progress.objects_failed, 0);
|
||||
assert_eq!(progress.skipped_objects, 1);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn test_recursive_bucket_heal_reports_typed_exhausted_and_permanent_failures() {
|
||||
let storage = Arc::new(MockStorage::default());
|
||||
|
||||
@@ -2320,9 +2320,13 @@ fn filter_policies_from_docs(policy_docs: &CacheEntity<PolicyDoc>, policy_name:
|
||||
}
|
||||
|
||||
fn build_group_desc(name: &str, group_info: GroupInfo, mapped_policy: Option<MappedPolicy>) -> GroupDesc {
|
||||
// A group without a mapped policy must report the group's own stored
|
||||
// timestamp, not the wall clock: this value feeds content-addressed
|
||||
// consumers (the site-replication repair plan hashes it), and a fresh
|
||||
// clock on every read makes equal states hash differently.
|
||||
let (policy, updated_at) = mapped_policy
|
||||
.map(|policy| (policy.policies, Some(policy.update_at)))
|
||||
.unwrap_or_else(|| (String::new(), Some(OffsetDateTime::now_utc())));
|
||||
.unwrap_or_else(|| (String::new(), group_info.update_at));
|
||||
|
||||
GroupDesc {
|
||||
name: name.to_string(),
|
||||
@@ -2347,6 +2351,46 @@ mod tests {
|
||||
};
|
||||
use tokio::sync::Notify;
|
||||
|
||||
#[test]
|
||||
fn test_build_group_desc_without_mapping_reports_the_stored_group_timestamp() {
|
||||
let stamp = OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("valid timestamp");
|
||||
let group_info = GroupInfo {
|
||||
version: 1,
|
||||
status: "enabled".to_string(),
|
||||
members: vec!["alice".to_string()],
|
||||
update_at: Some(stamp),
|
||||
};
|
||||
|
||||
let first = build_group_desc("team", group_info.clone(), None);
|
||||
let second = build_group_desc("team", group_info, None);
|
||||
|
||||
assert_eq!(first.updated_at, Some(stamp));
|
||||
assert_eq!(first.updated_at, second.updated_at);
|
||||
assert!(first.policy.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_build_group_desc_with_mapping_reports_the_mapping_timestamp() {
|
||||
let group_stamp = OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("valid timestamp");
|
||||
let mapping_stamp = OffsetDateTime::from_unix_timestamp(1_700_000_500).expect("valid timestamp");
|
||||
let group_info = GroupInfo {
|
||||
version: 1,
|
||||
status: "enabled".to_string(),
|
||||
members: vec![],
|
||||
update_at: Some(group_stamp),
|
||||
};
|
||||
let mapping = MappedPolicy {
|
||||
version: 1,
|
||||
policies: "readwrite".to_string(),
|
||||
update_at: mapping_stamp,
|
||||
};
|
||||
|
||||
let desc = build_group_desc("team", group_info, Some(mapping));
|
||||
|
||||
assert_eq!(desc.updated_at, Some(mapping_stamp));
|
||||
assert_eq!(desc.policy, "readwrite");
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
struct FailingInitialLoadStore;
|
||||
|
||||
|
||||
@@ -66,7 +66,7 @@ uuid = { workspace = true, features = ["v4", "serde", "fast-rng", "macro-diagnos
|
||||
[dev-dependencies]
|
||||
metrics-util = { workspace = true, features = ["debugging"] }
|
||||
proptest = "1"
|
||||
serial_test = { workspace = true }
|
||||
serial_test.workspace = true
|
||||
temp-env.workspace = true
|
||||
tokio = { workspace = true, features = ["macros", "fs", "rt-multi-thread"] }
|
||||
|
||||
|
||||
@@ -1568,6 +1568,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn abort_incomplete_multipart_upload_due_accepts_zero_days() {
|
||||
let initiated = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -1628,6 +1629,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn predict_expiration_selects_closest_expiry_for_put_object() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -1874,6 +1876,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn empty_transition_vectors_are_not_active_or_due() {
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
expiry_updated_at: None,
|
||||
@@ -1939,6 +1942,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn eval_inner_keeps_latest_object_before_days_due() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -1972,6 +1976,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn eval_inner_transitions_latest_object_after_days_due() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -2009,6 +2014,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn eval_inner_transitions_latest_object_after_date_due() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let transition_date = base_time - Duration::days(1);
|
||||
@@ -2048,6 +2054,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn eval_inner_selects_earliest_due_among_multiple_past_due_events() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
// Two enabled rules both yield a past-due DeleteAction and a third yields a
|
||||
@@ -2161,6 +2168,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn eval_inner_expires_noncurrent_version_after_due() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -2198,6 +2206,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn eval_inner_skips_noncurrent_expiration_without_successor() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("valid fixed test timestamp");
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -2233,6 +2242,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn eval_inner_missing_successor_does_not_skip_noncurrent_transition() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("valid fixed test timestamp");
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -2275,6 +2285,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn eval_inner_noncurrent_expiration_one_day_respects_due_boundary() {
|
||||
let successor_time = datetime!(2025-06-15 12:00:00 UTC);
|
||||
let due = expected_expiry_time(successor_time, 1);
|
||||
@@ -2316,6 +2327,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn eval_inner_expires_noncurrent_version_immediately_when_zero_days() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -2353,6 +2365,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn eval_inner_transitions_noncurrent_version_after_due() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -2428,6 +2441,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn evaluator_honors_newer_noncurrent_versions_retention_count() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let lc = Arc::new(BucketLifecycleConfiguration {
|
||||
@@ -2716,6 +2730,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn expired_object_delete_marker_ignores_marker_with_noncurrent_versions_present() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -2792,6 +2807,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn expired_object_delete_marker_deletes_only_delete_marker_immediately() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -2869,6 +2885,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn expiration_days_deletes_only_expired_delete_marker_when_due() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -2919,6 +2936,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn expiration_days_uses_earliest_due_rule_for_expired_delete_marker() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let make_rule = |id: &str, days| LifecycleRule {
|
||||
@@ -3249,6 +3267,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn del_marker_expiration_deletes_marker_and_older_versions_when_due() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("fixed timestamp should be valid");
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -3288,6 +3307,7 @@ mod tests {
|
||||
// --- TASK-003 tests: Round up to next UTC processing boundary ---
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn expected_expiry_time_rounds_up_to_next_midnight_utc() {
|
||||
with_default_ilm_process_time(|| {
|
||||
// Object created at 2025-01-15T10:30:45Z, expire in 30 days
|
||||
@@ -3303,6 +3323,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn expected_expiry_time_immediate_expiry_returns_epoch() {
|
||||
with_default_ilm_process_time(|| {
|
||||
let mod_time = datetime!(2025-06-01 12:00:00 UTC);
|
||||
@@ -3312,6 +3333,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn expected_expiry_time_preserves_exact_midnight_boundary() {
|
||||
with_default_ilm_process_time(|| {
|
||||
let mod_time = datetime!(2025-03-01 00:00:00 UTC);
|
||||
@@ -3321,6 +3343,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn expected_expiry_time_rounds_end_of_day_to_following_midnight() {
|
||||
with_default_ilm_process_time(|| {
|
||||
let mod_time = datetime!(2025-06-15 23:59:59 UTC);
|
||||
@@ -3330,6 +3353,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn expected_expiry_time_uses_canonical_process_time_boundary() {
|
||||
let mod_time = datetime!(2025-01-15 10:30:45 UTC);
|
||||
|
||||
@@ -3342,6 +3366,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn expected_expiry_time_uses_deprecated_process_time_alias() {
|
||||
let mod_time = datetime!(2025-01-15 10:30:45 UTC);
|
||||
|
||||
@@ -3354,6 +3379,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn expected_expiry_time_uses_default_boundary_when_process_time_is_zero_or_invalid() {
|
||||
let mod_time = datetime!(2025-01-15 10:30:45 UTC);
|
||||
|
||||
@@ -3376,6 +3402,7 @@ mod tests {
|
||||
|
||||
// (a) Default path (env unset) is byte-identical: one day == 86400s.
|
||||
#[test]
|
||||
#[serial]
|
||||
fn ilm_day_secs_defaults_to_86400_when_unset() {
|
||||
temp_env::with_var_unset(ENV_ILM_DEBUG_DAY_SECS, || {
|
||||
assert_eq!(ilm_day_secs(), DEFAULT_ILM_DAY_SECS);
|
||||
@@ -3404,6 +3431,7 @@ mod tests {
|
||||
|
||||
// (b) End-to-end env read scales the day length.
|
||||
#[test]
|
||||
#[serial]
|
||||
fn ilm_day_secs_scales_when_env_set() {
|
||||
temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("2"), || {
|
||||
assert_eq!(ilm_day_secs(), 2);
|
||||
@@ -3412,6 +3440,7 @@ mod tests {
|
||||
|
||||
// (c) Invalid env value falls back to 86400.
|
||||
#[test]
|
||||
#[serial]
|
||||
fn ilm_day_secs_falls_back_on_invalid_env() {
|
||||
temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("bogus"), || {
|
||||
assert_eq!(ilm_day_secs(), DEFAULT_ILM_DAY_SECS);
|
||||
@@ -3424,6 +3453,7 @@ mod tests {
|
||||
// Deadline math scales: with a 1s day and PROCESS_TIME unset, a Days=1 rule is
|
||||
// due 1s after mod_time (rounded up to the next 1s boundary => same instant).
|
||||
#[test]
|
||||
#[serial]
|
||||
fn expected_expiry_time_scales_with_debug_day_secs() {
|
||||
let mod_time = datetime!(2025-01-15 10:30:45 UTC);
|
||||
temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("1"), || {
|
||||
@@ -3439,6 +3469,7 @@ mod tests {
|
||||
|
||||
// days == 0 still yields the immediate-expiry sentinel regardless of the switch.
|
||||
#[test]
|
||||
#[serial]
|
||||
fn expected_expiry_time_zero_days_ignores_debug_day_secs() {
|
||||
let mod_time = datetime!(2025-06-01 12:00:00 UTC);
|
||||
temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("2"), || {
|
||||
@@ -3449,6 +3480,7 @@ mod tests {
|
||||
// (③) Interaction with an explicit RUSTFS_ILM_PROCESS_TIME: the deadline offset
|
||||
// uses the accelerated day length, but the rounding boundary honors PROCESS_TIME.
|
||||
#[test]
|
||||
#[serial]
|
||||
fn expected_expiry_time_debug_day_secs_respects_explicit_process_time() {
|
||||
let mod_time = datetime!(2025-01-15 10:30:00 UTC);
|
||||
// day == 10s, but round up to the next 60s (PROCESS_TIME) boundary.
|
||||
@@ -3465,6 +3497,7 @@ mod tests {
|
||||
|
||||
// (③) With the switch unset, an explicit PROCESS_TIME behaves exactly as before.
|
||||
#[test]
|
||||
#[serial]
|
||||
fn expected_expiry_time_unset_debug_day_secs_matches_legacy_process_time() {
|
||||
let mod_time = datetime!(2025-01-15 10:30:45 UTC);
|
||||
temp_env::with_var_unset(ENV_ILM_DEBUG_DAY_SECS, || {
|
||||
@@ -3492,6 +3525,7 @@ mod tests {
|
||||
|
||||
// The abort-incomplete-multipart deadline path also scales through the switch.
|
||||
#[test]
|
||||
#[serial]
|
||||
fn abort_incomplete_multipart_due_scales_with_debug_day_secs() {
|
||||
use s3s::dto::AbortIncompleteMultipartUpload;
|
||||
let initiated = datetime!(2025-01-15 10:30:45 UTC);
|
||||
@@ -3536,6 +3570,7 @@ mod tests {
|
||||
// (⑤ evaluator seam) A Days=1 rule fires under RUSTFS_ILM_DEBUG_DAY_SECS=1 once
|
||||
// `now` advances a few seconds past a mod_time only ~seconds in the past.
|
||||
#[test]
|
||||
#[serial]
|
||||
fn eval_inner_expires_days_one_rule_under_debug_day_secs() {
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
expiry_updated_at: None,
|
||||
@@ -3584,6 +3619,7 @@ mod tests {
|
||||
|
||||
// Absolute Date-based rules must NOT scale with the switch (regression guard).
|
||||
#[test]
|
||||
#[serial]
|
||||
fn eval_inner_date_rule_ignores_debug_day_secs() {
|
||||
let expiry_date = datetime!(2025-06-01 00:00:00 UTC);
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -3866,6 +3902,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn eval_inner_triggers_delete_all_versions_when_expired_object_all_versions_set() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -3904,6 +3941,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn expired_object_all_versions_does_not_apply_to_current_delete_marker() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("fixed timestamp should be valid");
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -3933,6 +3971,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn eval_inner_uses_delete_action_when_all_versions_not_set() {
|
||||
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
|
||||
let lc = BucketLifecycleConfiguration {
|
||||
@@ -4051,6 +4090,7 @@ mod tests {
|
||||
use super::*;
|
||||
use proptest::prelude::*;
|
||||
use s3s::dto::{NoncurrentVersionExpiration, Tag};
|
||||
use serial_test::serial;
|
||||
|
||||
const DAY_SECS: i64 = 86400;
|
||||
|
||||
@@ -4281,6 +4321,7 @@ mod tests {
|
||||
/// combination, and must be deterministic: the same input
|
||||
/// evaluated twice yields an identical event.
|
||||
#[test]
|
||||
#[serial]
|
||||
fn eval_inner_never_panics_and_is_deterministic(
|
||||
rules in prop::collection::vec(arb_rule(), 0..4),
|
||||
obj in arb_object_opts(),
|
||||
@@ -4420,6 +4461,7 @@ mod tests {
|
||||
/// candidate set — earliest due wins, ties prefer delete-class —
|
||||
/// and must be `NoneAction` exactly when that set is empty.
|
||||
#[test]
|
||||
#[serial]
|
||||
fn eval_inner_winner_matches_selection_oracle(
|
||||
rules in prop::collection::vec(arb_selection_rule(), 0..5),
|
||||
mod_off in 0i64..(2 * DAY_SECS),
|
||||
@@ -4473,6 +4515,7 @@ mod tests {
|
||||
/// non-decreasing in `days` (days == 0 maps to UNIX_EPOCH, below
|
||||
/// any post-1970 deadline).
|
||||
#[test]
|
||||
#[serial]
|
||||
fn expected_expiry_time_is_monotonic_in_days(
|
||||
mod_off in 0i64..(3650 * DAY_SECS),
|
||||
d1 in 0i32..2000,
|
||||
@@ -4494,6 +4537,7 @@ mod tests {
|
||||
/// to the next whole-day boundary: the result is day-aligned, not
|
||||
/// before `mod_time + days`, and less than one boundary beyond it.
|
||||
#[test]
|
||||
#[serial]
|
||||
fn expected_expiry_time_lands_on_default_day_boundary(
|
||||
mod_off in 0i64..(3650 * DAY_SECS),
|
||||
days in 1i32..2000,
|
||||
@@ -4511,6 +4555,7 @@ mod tests {
|
||||
/// to that boundary instead: aligned to it, never early, and less
|
||||
/// than one boundary late.
|
||||
#[test]
|
||||
#[serial]
|
||||
fn expected_expiry_time_lands_on_explicit_process_boundary(
|
||||
mod_off in 0i64..(365 * DAY_SECS),
|
||||
days in 1i32..400,
|
||||
|
||||
@@ -12,34 +12,10 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use std::collections::HashMap;
|
||||
//! Lifecycle tag decoding reuses the parser owned by `rustfs-replication`.
|
||||
//!
|
||||
//! `crates/lifecycle` already depends on `rustfs-replication`, so this is a
|
||||
//! plain re-export: no new crate edge, and no second copy of the parser to
|
||||
//! drift from the replication contract.
|
||||
|
||||
use url::form_urlencoded;
|
||||
|
||||
pub(crate) fn decode_tags_to_map(tags: &str) -> HashMap<String, String> {
|
||||
let mut list = HashMap::new();
|
||||
|
||||
for (k, v) in form_urlencoded::parse(tags.as_bytes()) {
|
||||
if k.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
list.insert(k.to_string(), v.to_string());
|
||||
}
|
||||
|
||||
list
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::decode_tags_to_map;
|
||||
|
||||
#[test]
|
||||
fn decode_tags_to_map_preserves_bucket_tagging_parser_behavior() {
|
||||
let tags = decode_tags_to_map("env=prod&encoded=a%2Fb&=ignored");
|
||||
|
||||
assert_eq!(tags.get("env").map(String::as_str), Some("prod"));
|
||||
assert_eq!(tags.get("encoded").map(String::as_str), Some("a/b"));
|
||||
assert!(!tags.contains_key(""));
|
||||
}
|
||||
}
|
||||
pub(crate) use rustfs_replication::tagging::decode_tags_to_map;
|
||||
|
||||
@@ -173,20 +173,28 @@ pub async fn lookup_config() -> Result<Args, OpaConfigError> {
|
||||
|
||||
impl AuthZPlugin {
|
||||
pub fn new(config: Args) -> Self {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(5))
|
||||
.connect_timeout(Duration::from_secs(1))
|
||||
.pool_max_idle_per_host(10)
|
||||
.pool_idle_timeout(Some(Duration::from_secs(60)))
|
||||
.tcp_keepalive(Some(Duration::from_secs(30)))
|
||||
.tcp_nodelay(true)
|
||||
.http2_keep_alive_interval(Some(Duration::from_secs(30)))
|
||||
.http2_keep_alive_timeout(Duration::from_secs(15))
|
||||
.build()
|
||||
.unwrap_or_else(|err| {
|
||||
error!("failed to build OPA HTTP client, falling back to default reqwest client: {err}");
|
||||
reqwest::Client::new()
|
||||
});
|
||||
let builder = || {
|
||||
reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(5))
|
||||
.connect_timeout(Duration::from_secs(1))
|
||||
.pool_max_idle_per_host(10)
|
||||
.pool_idle_timeout(Some(Duration::from_secs(60)))
|
||||
.tcp_keepalive(Some(Duration::from_secs(30)))
|
||||
.tcp_nodelay(true)
|
||||
.http2_keep_alive_interval(Some(Duration::from_secs(30)))
|
||||
.http2_keep_alive_timeout(Duration::from_secs(15))
|
||||
};
|
||||
// Never fall back to `reqwest::Client::new()`: it panics for the same
|
||||
// reason the first build failed (e.g. no system CA bundle, issue
|
||||
// #6734). Retry with an explicit empty trust store instead — an HTTP
|
||||
// OPA endpoint keeps working, an HTTPS one fails closed per request.
|
||||
let client = builder().build().unwrap_or_else(|err| {
|
||||
error!("failed to build OPA HTTP client ({err}); continuing with an empty trust store");
|
||||
builder()
|
||||
.tls_certs_only(std::iter::empty::<reqwest::Certificate>())
|
||||
.build()
|
||||
.expect("HTTP client construction must succeed with an explicit empty trust store")
|
||||
});
|
||||
|
||||
Self { client, args: config }
|
||||
}
|
||||
|
||||
@@ -52,4 +52,4 @@ It checks that:
|
||||
- KMS key ids are derived from each fixture's own `manifest.json`, so local static-KMS runs are not tied to one hard-coded key name
|
||||
- SSE-C `HEAD` responses round-trip the expected customer algorithm and customer-key MD5
|
||||
|
||||
These tests do not yet validate full plaintext reconstruction from MinIO-written encrypted data.
|
||||
These tests do not validate full plaintext reconstruction from MinIO-written encrypted data — that lives in the reader suite at `rustfs/src/storage/minio_generated_read_test.rs`, run with `--features rio-v2` over the same fixture captures.
|
||||
|
||||
@@ -471,4 +471,58 @@ mod tests {
|
||||
assert_eq!(parsed.delete_markers.len(), 1);
|
||||
assert_eq!(parsed.delete_markers[0].version_id, "marker-a");
|
||||
}
|
||||
|
||||
// Regression test for backlog#2076: a ListObjectsV2 response for a delimited
|
||||
// listing over a bucket that holds nested-key objects (e.g. any warm-tier
|
||||
// target that already stores more than one flat object) includes a
|
||||
// <CommonPrefixes><Prefix>...</Prefix></CommonPrefixes> element. `CommonPrefix`
|
||||
// previously had no `rename_all = "PascalCase"`, so quick_xml looked for a
|
||||
// lowercase `<prefix>` child, never found one, and (with no `#[serde(default)]`
|
||||
// either) failed the whole response with "missing field `prefix`" — surfacing to
|
||||
// callers of `WarmBackendS3::in_use()` (tier add/remove) as `TierPermErr`.
|
||||
#[test]
|
||||
fn list_objects_v2_xml_parses_common_prefixes() {
|
||||
let xml = br#"
|
||||
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
||||
<Name>warm-bucket</Name>
|
||||
<Prefix></Prefix>
|
||||
<Delimiter>/</Delimiter>
|
||||
<MaxKeys>1</MaxKeys>
|
||||
<IsTruncated>false</IsTruncated>
|
||||
<CommonPrefixes>
|
||||
<Prefix>subdir/</Prefix>
|
||||
</CommonPrefixes>
|
||||
</ListBucketResult>
|
||||
"#;
|
||||
|
||||
let parsed = quick_xml::de::from_reader::<_, ListBucketV2Result>(xml.as_slice()).expect("ListObjectsV2 XML should parse");
|
||||
|
||||
assert_eq!(parsed.common_prefixes.len(), 1);
|
||||
assert_eq!(parsed.common_prefixes[0].prefix, "subdir/");
|
||||
}
|
||||
|
||||
// Same fixture shape as list_object_versions_query hits (ListVersionsResult
|
||||
// reuses the same CommonPrefix type).
|
||||
#[test]
|
||||
fn list_object_versions_xml_parses_common_prefixes() {
|
||||
let xml = br#"
|
||||
<ListVersionsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
||||
<Name>warm-bucket</Name>
|
||||
<Prefix></Prefix>
|
||||
<KeyMarker></KeyMarker>
|
||||
<VersionIdMarker></VersionIdMarker>
|
||||
<MaxKeys>1</MaxKeys>
|
||||
<IsTruncated>false</IsTruncated>
|
||||
<CommonPrefixes>
|
||||
<Prefix>subdir/</Prefix>
|
||||
</CommonPrefixes>
|
||||
</ListVersionsResult>
|
||||
"#;
|
||||
|
||||
let parsed =
|
||||
quick_xml::de::from_reader::<_, ListVersionsResult>(xml.as_slice()).expect("ListObjectVersions XML should parse");
|
||||
|
||||
assert_eq!(parsed.common_prefixes.len(), 1);
|
||||
assert_eq!(parsed.common_prefixes[0].prefix, "subdir/");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,6 +24,7 @@ use std::{collections::HashMap, sync::Arc};
|
||||
use time::{Duration, OffsetDateTime, macros::format_description};
|
||||
use tracing::{error, info, warn};
|
||||
|
||||
use rustfs_utils::http::{is_amz_header, is_minio_header, is_rustfs_header, is_standard_header, is_storageclass_header};
|
||||
use s3s::dto::{ObjectLockLegalHoldStatus, ObjectLockRetentionMode, ReplicationStatus};
|
||||
use s3s::header::{
|
||||
X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, X_AMZ_REPLICATION_STATUS,
|
||||
@@ -40,7 +41,6 @@ use crate::{
|
||||
constants::{ISO8601_DATEFORMAT, MAX_MULTIPART_PUT_OBJECT_SIZE, MIN_PART_SIZE},
|
||||
credentials::SignatureType,
|
||||
transition_api::{ReaderImpl, TransitionClient, UploadInfo},
|
||||
utils::{is_amz_header, is_minio_header, is_rustfs_header, is_standard_header, is_storageclass_header},
|
||||
};
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
|
||||
@@ -30,6 +30,7 @@ use crate::utils::base64_decode;
|
||||
use super::transition_api;
|
||||
|
||||
#[derive(Debug, Default, Serialize, Deserialize)]
|
||||
#[serde(default, rename_all = "PascalCase")]
|
||||
pub struct CommonPrefix {
|
||||
pub prefix: String,
|
||||
}
|
||||
@@ -395,3 +396,30 @@ impl DeleteMultiObjects {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::ListBucketV2Result;
|
||||
|
||||
#[test]
|
||||
fn list_bucket_v2_common_prefix_accepts_s3_pascal_case_xml() {
|
||||
let xml = r#"
|
||||
<ListBucketResult>
|
||||
<Name>tier-bucket</Name>
|
||||
<Prefix></Prefix>
|
||||
<KeyCount>1</KeyCount>
|
||||
<MaxKeys>1000</MaxKeys>
|
||||
<Delimiter>/</Delimiter>
|
||||
<IsTruncated>false</IsTruncated>
|
||||
<CommonPrefixes>
|
||||
<Prefix>tenant-a/</Prefix>
|
||||
</CommonPrefixes>
|
||||
</ListBucketResult>
|
||||
"#;
|
||||
|
||||
let result = quick_xml::de::from_str::<ListBucketV2Result>(xml).expect("S3 list response should decode");
|
||||
|
||||
assert_eq!(result.common_prefixes.len(), 1);
|
||||
assert_eq!(result.common_prefixes[0].prefix, "tenant-a/");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -12,67 +12,6 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use lazy_static::lazy_static;
|
||||
use std::collections::HashMap;
|
||||
|
||||
use s3s::header::X_AMZ_STORAGE_CLASS;
|
||||
|
||||
lazy_static! {
|
||||
static ref SUPPORTED_QUERY_VALUES: HashMap<String, bool> = {
|
||||
let mut m = HashMap::new();
|
||||
m.insert("attributes".to_string(), true);
|
||||
m.insert("partNumber".to_string(), true);
|
||||
m.insert("versionId".to_string(), true);
|
||||
m.insert("response-cache-control".to_string(), true);
|
||||
m.insert("response-content-disposition".to_string(), true);
|
||||
m.insert("response-content-encoding".to_string(), true);
|
||||
m.insert("response-content-language".to_string(), true);
|
||||
m.insert("response-content-type".to_string(), true);
|
||||
m.insert("response-expires".to_string(), true);
|
||||
m
|
||||
};
|
||||
static ref SUPPORTED_HEADERS: HashMap<String, bool> = {
|
||||
let mut m = HashMap::new();
|
||||
m.insert("content-type".to_string(), true);
|
||||
m.insert("cache-control".to_string(), true);
|
||||
m.insert("content-encoding".to_string(), true);
|
||||
m.insert("content-disposition".to_string(), true);
|
||||
m.insert("content-language".to_string(), true);
|
||||
m.insert("x-amz-website-redirect-location".to_string(), true);
|
||||
m.insert("x-amz-object-lock-mode".to_string(), true);
|
||||
m.insert("x-amz-metadata-directive".to_string(), true);
|
||||
m.insert("x-amz-object-lock-retain-until-date".to_string(), true);
|
||||
m.insert("expires".to_string(), true);
|
||||
m.insert("x-amz-replication-status".to_string(), true);
|
||||
m
|
||||
};
|
||||
}
|
||||
|
||||
pub fn is_storageclass_header(header_key: &str) -> bool {
|
||||
header_key.to_lowercase() == X_AMZ_STORAGE_CLASS.as_str().to_lowercase()
|
||||
}
|
||||
|
||||
pub fn is_standard_header(header_key: &str) -> bool {
|
||||
*SUPPORTED_HEADERS.get(&header_key.to_lowercase()).unwrap_or(&false)
|
||||
}
|
||||
|
||||
pub fn is_amz_header(header_key: &str) -> bool {
|
||||
let key = header_key.to_lowercase();
|
||||
key.starts_with("x-amz-meta-")
|
||||
|| key.starts_with("x-amz-grant-")
|
||||
|| key == "x-amz-acl"
|
||||
|| rustfs_utils::http::is_sse_header(header_key)
|
||||
|| key.starts_with("x-amz-checksum-")
|
||||
}
|
||||
|
||||
pub fn is_rustfs_header(header_key: &str) -> bool {
|
||||
header_key.to_lowercase().starts_with("x-rustfs-")
|
||||
}
|
||||
|
||||
pub fn is_minio_header(header_key: &str) -> bool {
|
||||
header_key.to_lowercase().starts_with("x-minio-")
|
||||
}
|
||||
|
||||
/// Standard base64 (with `+`/`/` and `=` padding). Every base64 value this
|
||||
/// transition client emits or parses — `Content-MD5`, `x-amz-checksum-*`, and
|
||||
/// checksum digests in request/response bodies — is S3 wire format, which is
|
||||
|
||||
@@ -105,7 +105,8 @@ hex-simd.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
tracing-subscriber = { workspace = true, features = ["json", "env-filter", "time"] }
|
||||
temp-env = { workspace = true, features = ["async_closure"] }
|
||||
serial_test = { workspace = true }
|
||||
temp-env = { workspace = true }
|
||||
tempfile = { workspace = true }
|
||||
uuid = { workspace = true, features = ["v4", "serde", "fast-rng", "macro-diagnostics"] }
|
||||
tokio = { workspace = true, features = ["test-util", "fs", "rt-multi-thread"] }
|
||||
|
||||
@@ -951,8 +951,10 @@ impl ScannerConfigObjectDelete for SetDisks {
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use serial_test::serial;
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn runtime_tier_names_serves_cached_arc_within_ttl() {
|
||||
reset_tier_name_cache_for_test();
|
||||
// The tier config manager is unconfigured in unit tests, so the
|
||||
@@ -1028,6 +1030,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn foreground_read_guard_tracks_stream_lifetime() {
|
||||
reset_foreground_read_activity_for_test();
|
||||
assert_eq!(current_foreground_read_activity(), 0);
|
||||
@@ -1041,6 +1044,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn foreground_read_activity_keeps_larger_signal() {
|
||||
reset_foreground_read_activity_for_test();
|
||||
let _guard = ForegroundReadGuard::new();
|
||||
@@ -1053,6 +1057,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn scanner_runtime_guard_tracks_runtime_lifetime() {
|
||||
reset_scanner_runtime_instances_for_test();
|
||||
assert!(!scanner_runtime_initialized());
|
||||
|
||||
@@ -896,6 +896,7 @@ mod tests {
|
||||
SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE, SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION,
|
||||
SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE, SCANNER_SPEED, SCANNER_SUB_SYS, ScannerSpeed,
|
||||
};
|
||||
use serial_test::serial;
|
||||
use std::collections::HashMap;
|
||||
use std::time::Duration;
|
||||
use temp_env::{with_var, with_var_unset};
|
||||
@@ -943,6 +944,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn scanner_runtime_config_uses_persisted_values_when_env_is_unset() {
|
||||
let config = server_config_with_scanner(&[
|
||||
(SCANNER_SPEED, "slow"),
|
||||
@@ -1014,6 +1016,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn scanner_runtime_config_normalizes_persisted_default_speed() {
|
||||
let config = server_config_with_scanner(&[(SCANNER_SPEED, "default")]);
|
||||
|
||||
@@ -1029,6 +1032,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn scanner_runtime_config_prefers_env_over_persisted_config() {
|
||||
let config = server_config_with_scanner(&[(SCANNER_SPEED, "slowest"), (SCANNER_CYCLE, "600")]);
|
||||
|
||||
@@ -1045,6 +1049,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn scanner_runtime_config_prefers_heal_bitrot_cycle_over_scanner_compat_config() {
|
||||
let config = server_config_with_scanner_and_heal(&[(SCANNER_BITROT_CYCLE, "3600")], &[(HEAL_BITROT_CYCLE, "off")]);
|
||||
|
||||
@@ -1057,6 +1062,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn scanner_runtime_config_marks_scanner_bitrot_cycle_as_compat_source() {
|
||||
let config = server_config_with_scanner(&[(SCANNER_BITROT_CYCLE, "3600")]);
|
||||
|
||||
@@ -1073,6 +1079,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn scanner_runtime_config_normalizes_persisted_default_bitrot_cycles() {
|
||||
let default_cycle = DEFAULT_HEAL_BITROT_CYCLE_SECS.to_string();
|
||||
for config in [
|
||||
@@ -1097,6 +1104,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn scanner_runtime_config_validation_rejects_invalid_persisted_speed_with_env_override() {
|
||||
let config = server_config_with_scanner(&[(SCANNER_SPEED, "warp")]);
|
||||
|
||||
@@ -1130,6 +1138,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn scanner_runtime_config_uses_derived_delay_for_excessive_env_override() {
|
||||
let config = server_config_with_scanner(&[(SCANNER_SPEED, "slow")]);
|
||||
|
||||
@@ -1150,6 +1159,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn scanner_runtime_config_status_reports_value_sources() {
|
||||
let config = server_config_with_scanner(&[(SCANNER_CYCLE_MAX_OBJECTS, "100"), (SCANNER_CACHE_SAVE_TIMEOUT, "5")]);
|
||||
|
||||
@@ -1170,6 +1180,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn applied_runtime_config_is_the_authoritative_scheduler_state() {
|
||||
let config = server_config_with_scanner(&[(SCANNER_CYCLE, "321")]);
|
||||
|
||||
@@ -1186,6 +1197,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn scanner_runtime_config_status_reports_persisted_pacing_overrides() {
|
||||
let config = server_config_with_scanner(&[("delay", "3.5"), ("max_wait", "7")]);
|
||||
|
||||
@@ -1207,6 +1219,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn scanner_runtime_config_status_prefers_env_pacing_overrides() {
|
||||
let config = server_config_with_scanner(&[("delay", "3.5"), ("max_wait", "7")]);
|
||||
|
||||
@@ -1228,6 +1241,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn scanner_runtime_config_status_preserves_subsecond_max_wait() {
|
||||
let config = server_config_with_scanner(&[(SCANNER_SPEED, "fast")]);
|
||||
|
||||
|
||||
@@ -21,6 +21,7 @@ use crate::{
|
||||
ScannerPutObjReader as PutObjReader, init_bucket_metadata_sys_for_scanner_tests, init_ecstore_config_for_scanner_tests,
|
||||
init_local_disks_with_instance_ctx,
|
||||
};
|
||||
use serial_test::serial;
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::io::Cursor;
|
||||
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
|
||||
@@ -587,6 +588,7 @@ fn test_initial_scanner_delay_uses_configured_start_delay() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_initial_scanner_delay_uses_cycle_without_explicit_start_delay() {
|
||||
with_var(ENV_SCANNER_CYCLE, Some("120"), || {
|
||||
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
|
||||
@@ -633,6 +635,7 @@ fn test_initial_scanner_delay_keeps_delay_for_replication_without_buckets() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_scanner_cycle_max_duration_uses_env() {
|
||||
with_var(ENV_SCANNER_CYCLE_MAX_DURATION_SECS, Some("42"), || {
|
||||
assert_eq!(scanner_cycle_max_duration(), Some(Duration::from_secs(42)));
|
||||
@@ -676,6 +679,7 @@ async fn test_scanner_cycle_budget_drop_cancels_child_without_elapsed() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_scanner_cycle_budget_config_uses_work_budget_env() {
|
||||
with_var(ENV_SCANNER_CYCLE_MAX_OBJECTS, Some("100"), || {
|
||||
with_var(ENV_SCANNER_CYCLE_MAX_DIRECTORIES, Some("25"), || {
|
||||
@@ -687,6 +691,7 @@ fn test_scanner_cycle_budget_config_uses_work_budget_env() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_scanner_cycle_budget_config_disables_zero_work_budgets() {
|
||||
with_var(ENV_SCANNER_CYCLE_MAX_OBJECTS, Some("0"), || {
|
||||
with_var(ENV_SCANNER_CYCLE_MAX_DIRECTORIES, Some("0"), || {
|
||||
@@ -729,6 +734,7 @@ fn test_scan_cycle_partial_source_maps_budget_reason() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_mark_scan_cycle_idle_clears_published_cycle_state() {
|
||||
let mut cycle_info = CurrentCycle {
|
||||
current: 12,
|
||||
@@ -757,6 +763,7 @@ async fn test_mark_scan_cycle_idle_clears_published_cycle_state() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn scanner_cycle_metrics_guard_covers_published_first_cycle_lifetime() {
|
||||
let cycle_started = Utc::now() - chrono::Duration::seconds(5);
|
||||
let mut cycle_info = CurrentCycle {
|
||||
@@ -783,6 +790,7 @@ async fn scanner_cycle_metrics_guard_covers_published_first_cycle_lifetime() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn scanner_cycle_metrics_guard_keeps_active_cycle_published_during_finalization() {
|
||||
let mut cycle_info = CurrentCycle {
|
||||
current: 12,
|
||||
@@ -807,6 +815,7 @@ async fn scanner_cycle_metrics_guard_keeps_active_cycle_published_during_finaliz
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn scanner_cycle_metrics_guard_drop_clears_activity() {
|
||||
let guard = ScannerCycleMetricsGuard::new(CurrentCycle {
|
||||
current: 12,
|
||||
@@ -824,6 +833,7 @@ async fn scanner_cycle_metrics_guard_drop_clears_activity() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn run_data_scanner_cycle_publishes_activity_for_owner_lifetime() {
|
||||
let (_temp_dir, store) = setup_scanner_cycle_store().await;
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -874,6 +884,7 @@ async fn run_data_scanner_cycle_publishes_activity_for_owner_lifetime() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_finalize_partial_scan_cycle_advances_and_persists_counter() {
|
||||
let store = Arc::new(MemoryConfigStore::default());
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -909,6 +920,7 @@ async fn test_finalize_partial_scan_cycle_advances_and_persists_counter() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn scanner_cycle_recovers_to_newer_durable_cache_floor() {
|
||||
let store = Arc::new(MemoryConfigStore::default());
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -948,6 +960,7 @@ async fn scanner_cycle_recovers_to_newer_durable_cache_floor() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn scanner_cycle_rejects_invalid_cache_floor() {
|
||||
let store = Arc::new(MemoryConfigStore::default());
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -2369,6 +2382,7 @@ async fn scanner_usage_bootstrap_allows_first_bucket_to_win_startup() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn scanner_usage_backup_uses_durable_cycle_cadence_across_tasks() {
|
||||
let store = Arc::new(MemoryConfigStore::default());
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -2489,6 +2503,7 @@ fn scanner_cycle_advance_fails_before_reserved_exhausted_value() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_finalize_partial_scan_cycle_reports_persist_failure() {
|
||||
let store = Arc::new(MemoryConfigStore::default());
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -2512,6 +2527,7 @@ async fn test_finalize_partial_scan_cycle_reports_persist_failure() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_persist_scanner_cycle_state_reconciles_newer_winner() {
|
||||
let store = Arc::new(MemoryConfigStore::default());
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -3167,6 +3183,7 @@ async fn test_observational_usage_defers_when_authoritative_baseline_is_missing(
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_usage_route_barrier_precedes_durable_reconciliation() {
|
||||
let store = Arc::new(MemoryConfigStore::default());
|
||||
let key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_OBJ_NAME_PATH.as_str());
|
||||
@@ -3235,6 +3252,7 @@ async fn coordinator_does_not_put_after_remote_generation_flip() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_deferred_usage_save_keeps_last_real_save_metric() {
|
||||
let metrics = global_metrics();
|
||||
metrics.record_scanner_usage_save_result(ScannerUsageSaveResult::Success);
|
||||
@@ -4307,6 +4325,7 @@ fn scanner_cycle_cache_floor_stays_pending_during_deferred_usage_publication() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn finalizing_a_saved_cycle_acknowledges_its_exact_dirty_snapshot() {
|
||||
crate::scanner_io::clear_dirty_usage_bucket("photos");
|
||||
crate::scanner_io::record_dirty_usage_bucket("photos");
|
||||
@@ -4333,6 +4352,7 @@ fn finalizing_a_saved_cycle_acknowledges_its_exact_dirty_snapshot() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn finalizing_a_deferred_usage_save_keeps_dirty_work_pending() {
|
||||
crate::scanner_io::clear_dirty_usage_bucket("photos");
|
||||
crate::scanner_io::record_dirty_usage_bucket("photos");
|
||||
@@ -4375,6 +4395,7 @@ async fn scanner_cycle_keeps_remote_pending_acknowledgement() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn finalizing_an_already_durable_cycle_acknowledges_its_exact_dirty_snapshot() {
|
||||
crate::scanner_io::clear_dirty_usage_bucket("photos");
|
||||
crate::scanner_io::record_dirty_usage_bucket("photos");
|
||||
@@ -4389,6 +4410,7 @@ fn finalizing_an_already_durable_cycle_acknowledges_its_exact_dirty_snapshot() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn finalizing_a_prior_same_cycle_snapshot_keeps_new_dirty_work_pending() {
|
||||
crate::scanner_io::clear_dirty_usage_bucket("photos");
|
||||
crate::scanner_io::record_dirty_usage_bucket("photos");
|
||||
@@ -4404,6 +4426,7 @@ fn finalizing_a_prior_same_cycle_snapshot_keeps_new_dirty_work_pending() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn finalizing_a_durable_superseded_snapshot_keeps_dirty_work_pending() {
|
||||
crate::scanner_io::clear_dirty_usage_bucket("photos");
|
||||
crate::scanner_io::record_dirty_usage_bucket("photos");
|
||||
@@ -4419,6 +4442,7 @@ fn finalizing_a_durable_superseded_snapshot_keeps_dirty_work_pending() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn data_usage_persist_wait_covers_cache_retries_and_backup() {
|
||||
with_var(rustfs_config::ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, Some("7"), || {
|
||||
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
|
||||
@@ -4471,6 +4495,7 @@ async fn maintenance_feature_inspection_preserves_base_cycle_after_timeout() {
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
#[serial]
|
||||
async fn stable_maintenance_detection_preserves_base_cycle_after_timeout() {
|
||||
let ctx = CancellationToken::new();
|
||||
|
||||
@@ -4536,6 +4561,7 @@ async fn maintenance_feature_inspection_stops_on_cancellation() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_cycle_interval_prefers_explicit_cycle_override() {
|
||||
with_var(ENV_SCANNER_SPEED, Some("slowest"), || {
|
||||
with_var(ENV_SCANNER_CYCLE, Some("42"), || {
|
||||
@@ -4545,6 +4571,7 @@ fn test_cycle_interval_prefers_explicit_cycle_override() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_cycle_interval_prefers_explicit_cycle_over_default_cycle() {
|
||||
let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS);
|
||||
|
||||
@@ -4554,6 +4581,7 @@ fn test_cycle_interval_prefers_explicit_cycle_over_default_cycle() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_cycle_interval_uses_scanner_default_speed_override_when_unconfigured() {
|
||||
let _guard = ScannerDefaultSpeedGuard::set(ScannerSpeed::Slowest);
|
||||
|
||||
@@ -4563,6 +4591,7 @@ fn test_cycle_interval_uses_scanner_default_speed_override_when_unconfigured() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_cycle_interval_prefers_explicit_speed_over_default_speed_override() {
|
||||
let _guard = ScannerDefaultSpeedGuard::set(ScannerSpeed::Slowest);
|
||||
|
||||
@@ -4580,6 +4609,7 @@ fn test_cycle_interval_prefers_explicit_speed_over_default_speed_override() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_cycle_interval_uses_default_cycle_override_when_unconfigured() {
|
||||
let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS);
|
||||
|
||||
@@ -4761,6 +4791,7 @@ fn scanner_cycle_wait_plan_drives_growth_resets_and_bitrot_cap() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn scanner_cycle_schedule_status_reports_effective_backoff() {
|
||||
record_scanner_cycle_schedule(Duration::from_millis(86_400_001), true, 2_048, true, 7);
|
||||
|
||||
@@ -5077,6 +5108,7 @@ fn dirty_usage_wakes_are_disabled_for_explicit_cycle_policy() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn clean_idle_cap_preserves_default_bitrot_coverage_window() {
|
||||
let config = ScannerRuntimeConfig {
|
||||
bitrot_cycle: Some(Duration::from_secs(30 * 24 * 60 * 60)),
|
||||
@@ -5106,6 +5138,7 @@ fn clean_idle_cap_allows_policy_max_when_bitrot_is_disabled() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn clean_idle_cap_never_shortens_the_base_cycle() {
|
||||
let config = ScannerRuntimeConfig {
|
||||
bitrot_cycle: Some(Duration::from_secs(60)),
|
||||
@@ -5119,6 +5152,7 @@ fn clean_idle_cap_never_shortens_the_base_cycle() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_cycle_interval_keeps_default_cycle_with_explicit_speed() {
|
||||
let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS);
|
||||
|
||||
@@ -5136,6 +5170,7 @@ fn test_cycle_interval_keeps_default_cycle_with_explicit_speed() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_cycle_interval_prefers_explicit_start_delay_over_default_cycle() {
|
||||
let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS);
|
||||
|
||||
@@ -5149,6 +5184,7 @@ fn test_cycle_interval_prefers_explicit_start_delay_over_default_cycle() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_cycle_interval_supports_minio_speed_alias() {
|
||||
with_var_unset(ENV_SCANNER_SPEED, || {
|
||||
with_var_unset(ENV_SCANNER_CYCLE, || {
|
||||
@@ -5162,6 +5198,7 @@ fn test_cycle_interval_supports_minio_speed_alias() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_cycle_interval_supports_minio_cycle_alias() {
|
||||
with_var_unset(ENV_SCANNER_CYCLE, || {
|
||||
with_var_unset(ENV_SCANNER_START_DELAY_SECS, || {
|
||||
@@ -5181,6 +5218,7 @@ fn test_randomized_cycle_delay_handles_small_start_delay() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_wait_for_next_scanner_cycle_wakes_for_dirty_usage() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
|
||||
@@ -5206,6 +5244,7 @@ async fn test_wait_for_next_scanner_cycle_wakes_for_dirty_usage() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_wait_for_next_scanner_cycle_sees_unattempted_dirty_usage() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
let dirty_generation = crate::scanner_io::dirty_usage_generation();
|
||||
@@ -5227,6 +5266,7 @@ async fn test_wait_for_next_scanner_cycle_sees_unattempted_dirty_usage() {
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
#[serial]
|
||||
async fn test_wait_for_next_scanner_cycle_retries_stable_dirty_usage_on_timer() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
crate::scanner_io::record_dirty_usage_bucket("photos");
|
||||
@@ -5248,6 +5288,7 @@ async fn test_wait_for_next_scanner_cycle_retries_stable_dirty_usage_on_timer()
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
#[serial]
|
||||
async fn test_wait_for_next_scanner_cycle_can_defer_dirty_wakes_until_timer() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -5266,6 +5307,7 @@ async fn test_wait_for_next_scanner_cycle_can_defer_dirty_wakes_until_timer() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_wait_for_next_scanner_cycle_wakes_for_repeated_dirty_bucket() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
crate::scanner_io::record_dirty_usage_bucket("photos");
|
||||
@@ -5291,6 +5333,7 @@ async fn test_wait_for_next_scanner_cycle_wakes_for_repeated_dirty_bucket() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_wait_for_next_scanner_cycle_reschedules_for_runtime_config() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
let observed_generation = crate::runtime_config::scanner_runtime_config_generation();
|
||||
@@ -5318,6 +5361,7 @@ async fn test_wait_for_next_scanner_cycle_reschedules_for_runtime_config() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_wait_for_next_scanner_cycle_reschedules_for_maintenance_change() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
let observed_generation = crate::scanner_io::scanner_maintenance_generation();
|
||||
@@ -5629,6 +5673,7 @@ fn scanner_activity_after_a_cycle_restores_the_base_interval() {
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
#[serial]
|
||||
async fn distributed_clean_idle_wait_wakes_at_base_interval_for_remote_activity() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -5656,6 +5701,7 @@ async fn distributed_clean_idle_wait_wakes_at_base_interval_for_remote_activity(
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
#[serial]
|
||||
async fn superseded_retry_wait_defers_dirty_cluster_activity_until_timer() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -5751,6 +5797,7 @@ async fn superseded_retry_wait_wakes_when_remote_restart_clears_movement_state()
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
#[serial]
|
||||
async fn distributed_clean_idle_wait_blocks_backoff_for_unpropagated_maintenance() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -5777,6 +5824,7 @@ async fn distributed_clean_idle_wait_blocks_backoff_for_unpropagated_maintenance
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
#[serial]
|
||||
async fn distributed_clean_idle_wait_fails_closed_when_a_peer_is_unverifiable() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -5803,6 +5851,7 @@ async fn distributed_clean_idle_wait_fails_closed_when_a_peer_is_unverifiable()
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
#[serial]
|
||||
async fn distributed_clean_idle_wait_keeps_the_extended_deadline_when_peers_are_clean() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -5830,6 +5879,7 @@ async fn distributed_clean_idle_wait_keeps_the_extended_deadline_when_peers_are_
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
#[serial]
|
||||
async fn scanner_activity_probe_wait_is_cancellation_aware() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -5860,6 +5910,7 @@ async fn scanner_activity_probe_wait_is_cancellation_aware() {
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
#[serial]
|
||||
async fn scanner_activity_probe_wait_stops_after_leader_lock_loss() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
let ctx = CancellationToken::new();
|
||||
@@ -5891,6 +5942,7 @@ async fn scanner_activity_probe_wait_stops_after_leader_lock_loss() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_get_cycle_scan_mode_runs_deep_until_selection_window_completes() {
|
||||
with_var(ENV_SCANNER_BITROT_CYCLE_SECS, Some("3600"), || {
|
||||
let mode = get_cycle_scan_mode(10, 0, Some(Utc::now()), bitrot_scan_cycle());
|
||||
@@ -5899,6 +5951,7 @@ fn test_get_cycle_scan_mode_runs_deep_until_selection_window_completes() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_get_cycle_scan_mode_respects_elapsed_bitrot_cycle() {
|
||||
with_var(ENV_SCANNER_BITROT_CYCLE_SECS, Some("3600"), || {
|
||||
let recent = Utc::now() - chrono::Duration::minutes(30);
|
||||
@@ -5910,6 +5963,7 @@ fn test_get_cycle_scan_mode_respects_elapsed_bitrot_cycle() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_get_cycle_scan_mode_can_disable_periodic_deep_scan() {
|
||||
with_var(ENV_SCANNER_BITROT_CYCLE_SECS, Some("off"), || {
|
||||
assert_eq!(get_cycle_scan_mode(1, 0, None, bitrot_scan_cycle()), HealScanMode::Normal);
|
||||
@@ -5917,6 +5971,7 @@ fn test_get_cycle_scan_mode_can_disable_periodic_deep_scan() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_background_heal_info_for_scan_start_marks_deep_active() {
|
||||
let now = Utc::now();
|
||||
let info =
|
||||
@@ -5942,6 +5997,7 @@ fn background_heal_read_failures_never_become_initializable_defaults() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_background_heal_info_for_scan_start_keeps_deep_window_start() {
|
||||
with_var_unset(ENV_SCANNER_BITROT_CYCLE_SECS, || {
|
||||
let started_at = Utc::now();
|
||||
|
||||
@@ -18,6 +18,7 @@ use super::*;
|
||||
use crate::storage_api::VersionPurgeStatusType;
|
||||
use crate::{DiskOption, Endpoint, STORAGE_FORMAT_FILE, TierStats, new_disk, storageclass};
|
||||
use rustfs_filemeta::{FileInfo, FileMeta, MetadataResolutionParams};
|
||||
use serial_test::serial;
|
||||
use std::io::Write;
|
||||
#[cfg(unix)]
|
||||
use std::os::unix::fs::{PermissionsExt, symlink};
|
||||
@@ -375,6 +376,7 @@ impl Drop for TestGuard {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_should_skip_failed_respects_ttl() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir);
|
||||
@@ -396,6 +398,7 @@ async fn test_should_skip_failed_respects_ttl() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_record_failed_ttl_zero_noop() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(0, 100, &mut scanner, temp_dir);
|
||||
@@ -549,6 +552,7 @@ fn test_should_account_replication_stats_only_for_live_object_versions() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_heal_replication_only_queues_pending_null_deletes() {
|
||||
async fn replication_skipped_count() -> u64 {
|
||||
global_metrics()
|
||||
@@ -797,6 +801,7 @@ async fn test_scanner_heal_admission_accounting_maps_deep_scan_to_bitrot() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_excessive_version_alert_thresholds_use_env() {
|
||||
with_var(rustfs_config::ENV_SCANNER_ALERT_EXCESS_VERSIONS, Some("3"), || {
|
||||
with_var(rustfs_config::ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE, Some("100"), || {
|
||||
@@ -811,6 +816,7 @@ fn test_excessive_version_alert_thresholds_use_env() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_excessive_folders_threshold_uses_env() {
|
||||
with_var(rustfs_config::ENV_SCANNER_ALERT_EXCESS_FOLDERS, Some("3"), || {
|
||||
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
|
||||
@@ -820,6 +826,7 @@ fn test_excessive_folders_threshold_uses_env() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_excessive_folders_threshold_default_supports_pbs_layout() {
|
||||
with_var_unset(rustfs_config::ENV_SCANNER_ALERT_EXCESS_FOLDERS, || {
|
||||
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
|
||||
@@ -829,6 +836,7 @@ fn test_excessive_folders_threshold_default_supports_pbs_layout() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_scanner_yield_every_n_objects_uses_env() {
|
||||
with_var(rustfs_config::ENV_SCANNER_YIELD_EVERY_N_OBJECTS, Some("32"), || {
|
||||
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
|
||||
@@ -838,6 +846,7 @@ fn test_scanner_yield_every_n_objects_uses_env() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_scanner_yield_every_n_objects_uses_default() {
|
||||
with_var_unset(rustfs_config::ENV_SCANNER_YIELD_EVERY_N_OBJECTS, || {
|
||||
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
|
||||
@@ -964,6 +973,7 @@ fn test_order_folders_for_resume_reports_stale_hint() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_record_failed_prunes_to_max_entries() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(1000, 2, &mut scanner, temp_dir);
|
||||
@@ -995,6 +1005,7 @@ async fn test_record_failed_prunes_to_max_entries() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_prune_failed_objects_cache_drops_expired() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(5, 10, &mut scanner, temp_dir);
|
||||
@@ -1018,6 +1029,7 @@ async fn test_prune_failed_objects_cache_drops_expired() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_prune_failed_objects_max_zero_keeps_fresh() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(60, 0, &mut scanner, temp_dir);
|
||||
@@ -1840,6 +1852,7 @@ async fn test_heal_actions_returns_actual_size_without_inline_heal() {
|
||||
|
||||
#[tokio::test]
|
||||
#[cfg(unix)]
|
||||
#[serial]
|
||||
async fn test_scan_folder_skips_unreadable_child_directory() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(60, 0, &mut scanner, temp_dir.clone());
|
||||
@@ -1871,6 +1884,7 @@ async fn test_scan_folder_skips_unreadable_child_directory() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_scan_folder_exits_when_abandoned_child_listing_finishes() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
|
||||
@@ -1984,6 +1998,7 @@ async fn test_scan_folder_exits_when_abandoned_child_listing_finishes() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_scan_folder_xl_meta_named_directory_uses_namespace_descent() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
|
||||
@@ -2029,6 +2044,7 @@ async fn test_scan_folder_xl_meta_named_directory_uses_namespace_descent() {
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "current_thread")]
|
||||
#[serial]
|
||||
async fn test_scan_folder_corrupt_xl_meta_stops_erasure_data_dir_descent() {
|
||||
let logs = CapturedLogs::default();
|
||||
let subscriber = tracing_subscriber::fmt()
|
||||
@@ -2190,6 +2206,7 @@ async fn test_scan_folder_corrupt_xl_meta_stops_erasure_data_dir_descent() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_scan_folder_missing_xl_meta_stops_erasure_data_dir_descent() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
|
||||
@@ -2267,6 +2284,7 @@ async fn test_scan_folder_missing_xl_meta_stops_erasure_data_dir_descent() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_scan_folder_uuid_namespace_part_name_directory_is_not_data_dir() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
|
||||
@@ -2328,6 +2346,7 @@ async fn test_scan_folder_uuid_namespace_part_name_directory_is_not_data_dir() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_scan_folder_non_erasure_metadata_keeps_namespace_descent() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
|
||||
@@ -2369,6 +2388,7 @@ async fn test_scan_folder_non_erasure_metadata_keeps_namespace_descent() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_scan_folder_compacted_parent_sends_partial_update() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
|
||||
@@ -2410,6 +2430,7 @@ async fn test_scan_folder_compacted_parent_sends_partial_update() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_scan_data_folder_cancelled_before_scan_clears_current_path() {
|
||||
let (scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard {
|
||||
@@ -2454,6 +2475,7 @@ async fn test_scan_data_folder_cancelled_before_scan_clears_current_path() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_scan_data_folder_returns_partial_cache_on_budget_cancel() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
|
||||
@@ -2509,6 +2531,7 @@ async fn test_scan_data_folder_returns_partial_cache_on_budget_cancel() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_scan_data_folder_reports_invalid_checkpoint_ignored_once() {
|
||||
let (scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard {
|
||||
@@ -2553,6 +2576,7 @@ async fn test_scan_data_folder_reports_invalid_checkpoint_ignored_once() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_scan_data_folder_resume_hint_prioritizes_next_existing_folder() {
|
||||
let (scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard {
|
||||
@@ -2626,6 +2650,7 @@ async fn test_scan_data_folder_resume_hint_prioritizes_next_existing_folder() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn scan_data_folder_missing_bucket_returns_partial() {
|
||||
let (scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard {
|
||||
@@ -2677,6 +2702,7 @@ async fn scan_data_folder_missing_bucket_returns_partial() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn scan_data_folder_missing_scan_root_returns_partial() {
|
||||
let (scanner, temp_dir) = build_test_scanner().await;
|
||||
tokio::fs::remove_dir_all(&temp_dir)
|
||||
@@ -2722,6 +2748,7 @@ async fn scan_data_folder_missing_scan_root_returns_partial() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_scan_data_folder_resume_hint_orders_across_new_and_existing_folders() {
|
||||
let (scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard {
|
||||
@@ -2790,6 +2817,7 @@ async fn test_scan_data_folder_resume_hint_orders_across_new_and_existing_folder
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_scan_data_folder_partial_object_budget_accumulates_progress() {
|
||||
let (scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard {
|
||||
@@ -2872,6 +2900,7 @@ async fn test_scan_data_folder_partial_object_budget_accumulates_progress() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_partial_compacted_entry_does_not_carry_children() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard {
|
||||
@@ -2917,6 +2946,7 @@ async fn test_partial_compacted_entry_does_not_carry_children() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_partial_entry_does_not_carry_missing_old_child() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard {
|
||||
@@ -2949,6 +2979,7 @@ async fn test_partial_entry_does_not_carry_missing_old_child() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_legacy_windows_cache_rebuilds_and_round_trips_portable_keys() {
|
||||
let (scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard {
|
||||
@@ -3015,6 +3046,7 @@ async fn test_legacy_windows_cache_rebuilds_and_round_trips_portable_keys() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_scan_data_folder_success_clears_resume_hint() {
|
||||
let (scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard {
|
||||
@@ -3057,6 +3089,7 @@ async fn test_scan_data_folder_success_clears_resume_hint() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_scan_data_folder_keeps_unresolved_objects_partial() {
|
||||
let (scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard {
|
||||
@@ -3104,6 +3137,7 @@ async fn test_scan_data_folder_keeps_unresolved_objects_partial() {
|
||||
|
||||
#[tokio::test]
|
||||
#[cfg(unix)]
|
||||
#[serial]
|
||||
async fn test_scan_folder_ignores_symlinked_child_directory() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(60, 0, &mut scanner, temp_dir.clone());
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user