fran/rustfs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,803 Commits 52 Branches 132 Tags
f5d7fea7a41ea6948c0d47df6e13f0afe2bb178e
Commit Graph
3803 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Zhengchao AnandGitHub f5d7fea7a4 fix(ecstore): remove unused mirror fields (#4019) 2026-06-28 22:31:37 +08:00
Zhengchao AnandGitHub 0f3213a530 docs: update security advisory lessons (#4018) 2026-06-28 21:55:40 +08:00
Zhengchao AnandGitHub 99f981a868 docs: plan ecstore split boundaries (#4017) 
docs: plan ecstore module split boundaries
 2026-06-28 21:55:10 +08:00
Zhengchao AnandGitHub ed0f191e1a docs: document global singleton migration plan (#730) (#4015) 
docs: document global singleton migration plan

Add comments documenting the Tier A/B classification and migration
plan for global singletons in ecstore runtime.

Refs #730
 2026-06-28 21:25:42 +08:00
Zhengchao AnandGitHub 163e8089db fix: remove all backlog links from code and comments (#4013) 2026-06-28 20:55:25 +08:00
Zhengchao AnandGitHub aa8b8b5706 docs: document crypto RC version dependencies (#732) (#4011) 
* docs: document crypto RC version dependencies

Add comment explaining why aes-gcm and chacha20poly1305 use RC
versions and the migration path when stable versions are released.

Refs #732

* fix: remove all backlog links from code and comments
 2026-06-28 20:47:51 +08:00
Zhengchao AnGitHubhouseme
 		cd165ab181 docs: document obs reverse dependency on ecstore (#735) (#4010) 
docs: document obs reverse dependency on ecstore

Add comment explaining why obs depends on ecstore and the scope
of work required to break this dependency.

Refs #735

Co-authored-by: houseme <[email protected]>
 2026-06-28 20:28:04 +08:00
Zhengchao AnandGitHub a37e918936 docs: document SHA-1 HMAC migration proposal (#747) (#4009) 2026-06-28 19:42:18 +08:00
Zhengchao AnGitHubhouseme
 		8d0ba1cd3c docs: document deadlock detector mutex design rationale (#744) (#4008) 
docs: document deadlock detector mutex design rationale

Add comment explaining why std::sync::Mutex is used instead of
tokio::sync::Mutex in the deadlock detector.

Refs #744

Co-authored-by: houseme <[email protected]>
 2026-06-28 19:29:50 +08:00
Zhengchao AnandGitHub 27b8592879 docs: add documentation to storage-api public types (#741) (#4007) 
docs: add documentation to storage-api public types

Add doc comments to public structs, enums, and traits in
storage-api crate to improve documentation coverage.

Refs #741
 2026-06-28 19:12:02 +08:00
Zhengchao AnandGitHub 1f56ddf913 docs: document table_catalog mutex design rationale (#739) (#4006) 
docs: document table_catalog mutex design rationale

Add comment explaining why the single mutex is intentional and
what alternatives to consider if contention becomes a bottleneck.

Refs #739
 2026-06-28 18:56:13 +08:00
Zhengchao AnandGitHub 710ae74cde perf: add S3 operations benchmark framework (#738) (#4005) 2026-06-28 18:02:41 +08:00
Zhengchao AnandGitHub 51acf2a99c feat: add rate limiting middleware framework (#737) (#4002) 
* feat: add rate limiting middleware framework

Add rate limiting middleware with token bucket algorithm for
per-client request rate limiting. This provides the foundation
for DoS protection.

Refs #737

* fix: address clippy lints in rate_limit.rs

- Collapse nested if statements into single if-let chains
- Use .is_multiple_of() instead of manual modulo check
 2026-06-28 17:48:40 +08:00
Zhengchao AnandGitHub ee82d6c026 test: add insta snapshot test for storage error display format (#740) (#4001) 
test: add insta snapshot test for storage error display format

Add snapshot test to detect unexpected changes in StorageError
display format. This catches output format regressions that
traditional assert tests might miss.

Refs #740
 2026-06-28 16:10:07 +08:00
Zhengchao AnandGitHub 84cdf12083 test(security): add security boundary tests (#748) (#3998) 
test(security): add security boundary tests

Add e2e tests for security-sensitive scenarios:
- Large XML body handling (DoS protection)
- Excessive multipart parts (DoS protection)
- Concurrent object operations (race condition handling)
- Internal URL validation (SSRF prevention)

Refs #748
 2026-06-28 15:17:30 +08:00
Zhengchao AnandGitHub c768a9c382 docs(storage-api): document filemeta dependency as known limitation (#731) (#3997) 
* docs(storage-api): document filemeta dependency as known limitation

Add comment explaining why storage-api depends on filemeta and
the scope of work required to break this dependency (300+ files).

Refs https://github.com/rustfs/backlog/issues/731

* docs(storage-api): remove backlog link from comment
 2026-06-28 14:59:24 +08:00
Zhengchao AnandGitHub c475d03b48 fix: replace unwrap() with expect() in more files (#729 batch 14) (#3994) 2026-06-28 11:45:23 +08:00
Zhengchao AnandGitHub 05d201679c fix: replace unwrap() with expect() in more files (#729 batch 13) (#3993) 2026-06-28 11:45:13 +08:00
Zhengchao AnandGitHub f0ab812213 fix: replace unwrap() with expect() in remaining files (#729 batch 12) (#3992) 2026-06-28 11:45:03 +08:00
Zhengchao AnandGitHub 1f8a5bc095 fix(ecstore): replace unwrap() with proper error handling in api_get_object_attributes (#729 batch 11) (#3991) 
fix(ecstore): replace unwrap() with proper error handling in api_get_object_attributes

Replace unsafe unwrap() calls with proper error handling in
api_get_object_attributes.rs:
- HTTP header access now uses ok_or_else with descriptive messages
- String parsing now uses map_err with descriptive messages
- HeaderValue creation now uses expect with descriptive messages

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:23:41 +08:00
Zhengchao AnandGitHub 25170943ce fix(ecstore): improve expect() messages in admin_server_info (#729 batch 9) (#3990) 
fix(ecstore): improve expect() messages in admin_server_info

Replace unwrap() with expect() for better error diagnostics in
admin_server_info.rs:
- URL host/port access now has descriptive messages
- HashMap get_mut calls now have descriptive messages

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:23:04 +08:00
Zhengchao AnandGitHub 20c7cb1074 fix(server): improve expect() messages in layer.rs (#729 batch 8) (#3989) 
fix(server): improve expect() messages in layer.rs

Replace unwrap() with expect(valid response body) for Response
builder calls in layer.rs.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:22:46 +08:00
Zhengchao AnandGitHub 5c94fe7dd7 fix(ecstore): improve expect() messages in replication_resyncer (#729 batch 7) (#3988) 
fix(ecstore): improve expect() messages in replication_resyncer

Replace unwrap() with expect() for better error diagnostics in
replication_resyncer.rs:
- HashMap get_mut calls now have descriptive expect messages
- format() calls now use unwrap_or_else for error handling

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:22:26 +08:00
Zhengchao AnandGitHub d668ef837a fix(admin): improve expect() messages in remaining admin handlers (#729 batch 6) (#3987) 
fix(admin): improve expect() messages in remaining admin handlers

Replace generic parse().unwrap() with parse().expect(valid header value)
for better error diagnostics in service_account.rs, group.rs, and console.rs.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:22:08 +08:00
Zhengchao AnandGitHub e5a355efe9 fix(admin): improve expect() messages in policies handler (#729 batch 5) (#3986) 
fix(admin): improve expect() messages in policies handler

Replace generic parse().unwrap() with parse().expect(valid header value)
for better error diagnostics in policies.rs.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:21:51 +08:00
Zhengchao AnandGitHub dbca96faf6 fix(admin): improve expect() messages in user handler (#729 batch 4) (#3985) 
fix(admin): improve expect() messages in user handler

Replace generic parse().unwrap() with parse().expect(valid header value)
for better error diagnostics in user.rs.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:21:24 +08:00
Zhengchao AnandGitHub e8470bbc66 fix(admin): replace unwrap() with safe pattern in bucket_meta handler (#729 batch 3) (#3984) 
fix(admin): replace unwrap() with safe pattern in bucket_meta handler

Replace 11 instances of HashMap.get_mut().unwrap() with
match pattern that continues to next iteration if key is missing.

Also improve expect() messages for header value parsing.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:20:51 +08:00
housemeGitHubheihutu
 		46d7f9e1f2 feat(get): harden codec streaming rollout (#3981) 
* feat(get): consolidate GET performance optimization

Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.

## Changes

### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support

### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
  - delete_marker_votes tracking
  - requested_version_id and matching_version_votes tracking
  - version_early_stop_decision() method
- 6 new tests for version early-stop scenarios

### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default

### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration

### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)

### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label

### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants

### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs

### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |

## Rollback

All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false

Co-Authored-By: heihutu <[email protected]>

* test(get): add stress test scripts for GET optimization validation

- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation

Co-Authored-By: heihutu <[email protected]>

* test(ecstore): align file cache reclaim defaults

* chore(deps): update redis and erasure codec

* test(ecstore): align decode fill policy default

* fix(get): wire codec streaming rollout gate

* perf(get): skip metrics-off codec timers

* test(get): capture codec streaming diagnostics

* test(get): add multipart fallback probe

* test(get): add encrypted fallback probe

* test(get): add compressed fallback probe

* test(get): add degraded read fallback probe

* test(get): cover remote fallback probe

* test(get): report warp request p99

* test(get): capture OTLP metric deltas

* perf(get): align codec streaming inflight default

* perf(get): reuse codec reader output buffers

* test(get): count codec reader fill starts

* perf(get): reuse codec reader fill worker

* perf(get): lazy init rustfs codec reconstruct

* test(get): cover rustfs codec source faults

* docs(get): record rustfs codec fallback scope

* feat(get): add multipart codec reader opt-in

* test(get): add multipart codec smoke option

* test(get): cover multipart codec degraded fallback

* perf(get): bound multipart codec eager setup

* test(get): satisfy codec hardening PR gate

---------

Co-authored-by: heihutu <[email protected]>
 2026-06-28 11:20:21 +08:00
Zhengchao AnandGitHub d99056902e fix(admin): replace unwrap() with proper error handling in tier handler (#729 batch 2) (#3983) 
fix(admin): replace unwrap() with proper error handling in tier handler

Replace 9 instances of args.{type}.clone().unwrap() with
ok_or_else() that returns a descriptive S3Error when the
tier configuration is missing.

Also improve expect() messages for header value parsing.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:00:50 +08:00
Zhengchao AnandGitHub 6e72dc3076 fix(ecstore): replace unsafe k.unwrap() in bucket_target_sys (#729) (#3982) 
fix(ecstore): replace k.unwrap() with safe pattern in bucket_target_sys

Replace unsafe k.unwrap().as_str() with if let Some(key_str) pattern
in 5 locations where HeaderMap iterator yields (Option<HeaderName>, Value).

This prevents potential panics if header names are invalid.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 10:42:56 +08:00
Zhengchao AnandGitHub 259a99a501 fix(ci): pin protocol matrix checkout action (#3977) 2026-06-28 10:04:36 +08:00
housemeandGitHub c928fd1c7a chore(deps): update flake.lock (#3978) 2026-06-28 09:54:22 +08:00
Zhengchao AnandGitHub 7238a937a9 fix: correct misleading zero-copy/direct-io docs and internal naming (#733 Phase 1) (#3976) 2026-06-28 09:12:49 +08:00
Zhengchao AnandGitHub 3c6dc2a633 ci: remove --no-default-features from protocol test matrix (#3980) 2026-06-28 09:12:23 +08:00
Zhengchao AnandGitHub e1272f2aba revert: restore #![allow(dead_code)] - CI clippy -D warnings conflict (#3979) 
revert: restore #![allow(dead_code)] - clippy -D warnings treats warn as error

The #742 PR changed #![allow(dead_code)] to #![warn(dead_code)], but
CI runs clippy with -D warnings which turns warnings into errors.
This caused CI failures across multiple PRs.

Reverting to #![allow(dead_code)] until the dead code is actually
cleaned up. The 189 warnings in ecstore should be fixed incrementally
by deleting dead code and adding item-level allows, not by changing
the crate-level policy.
 2026-06-28 08:32:34 +08:00
Henry GuoGitHubHenry Guohouseme
 		1e303e5be0 fix(data-usage): refresh versioned usage state (#3969) 
fix(data-usage): refresh versioned usage from authoritative state

Co-authored-by: Henry Guo <[email protected]>
Co-authored-by: houseme <[email protected]>
 2026-06-28 07:58:07 +08:00
Zhengchao AnandGitHub 129ad2296b ci: add protocol feature test matrix (swift, sftp, no-default) (#736) (#3975) 2026-06-28 07:51:15 +08:00
Zhengchao AnandGitHub 113058af54 chore: replace blanket #![allow(dead_code)] with #![warn(dead_code)] (#742) (#3974) 2026-06-28 07:50:51 +08:00
housemeGitHubheihutu
 		27468ebfa9 feat(get): consolidate GET performance optimization (#3972) 
* feat(get): consolidate GET performance optimization

Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.

## Changes

### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support

### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
  - delete_marker_votes tracking
  - requested_version_id and matching_version_votes tracking
  - version_early_stop_decision() method
- 6 new tests for version early-stop scenarios

### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default

### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration

### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)

### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label

### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants

### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs

### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |

## Rollback

All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false

Co-Authored-By: heihutu <[email protected]>

* test(get): add stress test scripts for GET optimization validation

- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation

Co-Authored-By: heihutu <[email protected]>

* test(ecstore): align file cache reclaim defaults

* chore(deps): update redis and erasure codec

* test(ecstore): align decode fill policy default

* test(ecstore): align metadata early-stop default

* fix(ecstore): keep metadata early stop opt-in

---------

Co-authored-by: heihutu <[email protected]>
 2026-06-28 07:14:07 +08:00
Zhengchao AnGitHubhouseme
 		512418cda9 fix(ecstore): replace unbounded metadata cache with moka (#743) (#3970) 
fix(ecstore): replace unbounded metadata cache with moka

Replace the manual Arc<RwLock<HashMap>> metadata cache with
moka::future::Cache, which provides:
- Built-in LRU eviction when max_capacity is reached
- Automatic TTL expiry via time_to_live (250ms)
- Lock-free concurrent reads
- Non-blocking invalidation

Fixes the memory leak risk from unbounded HashMap and the
all-or-nothing eviction logic that cleared all entries at once.

Closes #743

Co-authored-by: houseme <[email protected]>
 2026-06-28 03:01:32 +08:00
housemeandGitHub 175566f037 docs(get): record default switch readiness (#3967) 
* docs(get): record default switch readiness

* chore(docs): keep pr35 evaluation note local

* refactor(bench): trim pr35 readiness python
 2026-06-28 01:12:44 +08:00
Zhengchao AnandGitHub 7ae9697d42 docs: close final architecture audit gaps (#3966) 2026-06-28 00:07:42 +08:00
Zhengchao AnandGitHub 79234c030d docs: close architecture migration validation (#3965) 2026-06-27 23:42:51 +08:00
Zhengchao AnandGitHub 82bbef0b60 test: cover runtime and repair preservation (#3964) 2026-06-27 23:34:59 +08:00
housemeandGitHub bf03ff2869 feat(get): add limited opt-in rollout gates (#3963) 
* feat(bench): harden cooled get ab harness

* feat(get): add limited opt-in rollout gates

* fix(get): tighten rollout gate fallbacks
 2026-06-27 23:13:02 +08:00
Zhengchao AnandGitHub 66ad138505 docs: close phase 7 global split plan (#3962) 2026-06-27 22:25:16 +08:00
housemeandGitHub 47d05a1d6a feat(bench): harden cooled get ab harness (#3960) 2026-06-27 22:23:09 +08:00
Zhengchao AnandGitHub c7dc3d7974 refactor: move runtime fallback defaults to owners (#3961) 2026-06-27 22:21:09 +08:00
housemeandGitHub 20f56af09c feat(get): tune output response handoff (#3956) 2026-06-27 21:46:38 +08:00
Zhengchao AnandGitHub 68d5d1d41d refactor: isolate specialized runtime fallbacks (#3958) 2026-06-27 21:35:47 +08:00