copilot-swe-agent[bot] and houseme
90f8178da3
docs: add comprehensive documentation and tests for concurrent GetObject optimization
...
- Add detailed technical documentation explaining the solution
- Document root cause analysis and solution architecture
- Include performance expectations and testing recommendations
- Add integration tests for concurrency tracking and buffer sizing
- Add cache behavior tests
- Include benchmark tests for concurrent request handling
Co-authored-by: houseme <[email protected]
2025-11-24 03:49:51 +00:00
copilot-swe-agent[bot] and houseme
901e5b4e0c
feat: add concurrency-aware buffer sizing and hot object caching for GetObject
...
- Implement adaptive buffer sizing based on concurrent request load
- Add per-request tracking with automatic cleanup using RAII guards
- Implement hot object cache (LRU) for frequently accessed small files (<= 10MB)
- Add disk I/O semaphore to prevent saturation under extreme load
- Integrate concurrency module into GetObject implementation
- Buffer sizes now adapt: low concurrency uses large buffers for throughput,
high concurrency uses smaller buffers for fairness and memory efficiency
- Add comprehensive metrics collection for monitoring performance
Co-authored-by: houseme <[email protected]
2025-11-24 03:46:21 +00:00
copilot-swe-agent[bot]
7ce91a0a94
Initial plan
2025-11-24 03:33:54 +00:00
44bdebe6e9
build(deps): bump the dependencies group with 10 updates ( #914 )
...
* build(deps): bump the dependencies group with 10 updates
* build(deps): bump the dependencies group with 8 updates (#913 )
Bumps the dependencies group with 8 updates:
| Package | From | To |
| --- | --- | --- |
| [bytesize](https://github.com/bytesize-rs/bytesize ) | `2.2.0` | `2.3.0` |
| [aws-config](https://github.com/smithy-lang/smithy-rs ) | `1.8.10` | `1.8.11` |
| [aws-credential-types](https://github.com/smithy-lang/smithy-rs ) | `1.2.9` | `1.2.10` |
| [aws-sdk-s3](https://github.com/awslabs/aws-sdk-rust ) | `1.113.0` | `1.115.0` |
| [convert_case](https://github.com/rutrum/convert-case ) | `0.9.0` | `0.10.0` |
| [hashbrown](https://github.com/rust-lang/hashbrown ) | `0.16.0` | `0.16.1` |
| [rumqttc](https://github.com/bytebeamio/rumqtt ) | `0.25.0` | `0.25.1` |
| [starshard](https://github.com/houseme/starshard ) | `0.5.0` | `0.6.0` |
Updates `bytesize` from 2.2.0 to 2.3.0
- [Release notes](https://github.com/bytesize-rs/bytesize/releases )
- [Changelog](https://github.com/bytesize-rs/bytesize/blob/master/CHANGELOG.md )
- [Commits](https://github.com/bytesize-rs/bytesize/compare/bytesize-v2.2.0...bytesize-v2.3.0 )
Updates `aws-config` from 1.8.10 to 1.8.11
- [Release notes](https://github.com/smithy-lang/smithy-rs/releases )
- [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/smithy-lang/smithy-rs/commits )
Updates `aws-credential-types` from 1.2.9 to 1.2.10
- [Release notes](https://github.com/smithy-lang/smithy-rs/releases )
- [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/smithy-lang/smithy-rs/commits )
Updates `aws-sdk-s3` from 1.113.0 to 1.115.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases )
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits )
Updates `convert_case` from 0.9.0 to 0.10.0
- [Commits](https://github.com/rutrum/convert-case/commits )
Updates `hashbrown` from 0.16.0 to 0.16.1
- [Release notes](https://github.com/rust-lang/hashbrown/releases )
- [Changelog](https://github.com/rust-lang/hashbrown/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/hashbrown/compare/v0.16.0...v0.16.1 )
Updates `rumqttc` from 0.25.0 to 0.25.1
- [Release notes](https://github.com/bytebeamio/rumqtt/releases )
- [Changelog](https://github.com/bytebeamio/rumqtt/blob/main/CHANGELOG.md )
- [Commits](https://github.com/bytebeamio/rumqtt/compare/rumqttc-0.25.0...rumqttc-0.25.1 )
Updates `starshard` from 0.5.0 to 0.6.0
- [Commits](https://github.com/houseme/starshard/compare/0.5.0...0.6.0 )
---
updated-dependencies:
- dependency-name: bytesize
dependency-version: 2.3.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: dependencies
- dependency-name: aws-config
dependency-version: 1.8.11
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
- dependency-name: aws-credential-types
dependency-version: 1.2.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
- dependency-name: aws-sdk-s3
dependency-version: 1.115.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: dependencies
- dependency-name: convert_case
dependency-version: 0.10.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: dependencies
- dependency-name: hashbrown
dependency-version: 0.16.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
- dependency-name: rumqttc
dependency-version: 0.25.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
- dependency-name: starshard
dependency-version: 0.6.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: dependencies
...
Signed-off-by: dependabot[bot] <[email protected] [email protected] [email protected]
2025-11-24 11:03:35 +08:00
majinghe and GitHub
2b268fdd7f
update tls configuration in helm chart ( #900 )
...
* update tls configuration in helm chart
* typo fix
2025-11-20 22:20:11 +08:00
houseme and GitHub
18cd9a8b46
build(deps): bump the dependencies group with 5 updates ( #896 )
2025-11-20 13:04:24 +08:00
loverustfs and GitHub
e14809ee04
Revise data sovereignty and compliance details in README
...
Updated the comparison between RustFS and other object storage solutions to clarify data sovereignty and compliance aspects.
2025-11-20 09:11:15 +08:00
loverustfs and GitHub
390d051ddd
Update README.md
...
Correcting inaccurate expressions
2025-11-20 08:55:14 +08:00
b73ca0220e
add ifmatch support ( #885 )
...
Co-authored-by: loverustfs <[email protected]
2025-11-20 08:40:47 +08:00
loverustfs
b4071d493c
Modify README
2025-11-20 08:24:07 +08:00
277d80de13
Fix: Implement priority-based heal queue with comprehensive diagnostic logging ( #884 )
...
* Initial plan
* Implement priority-based heal queue with deduplication
Co-authored-by: houseme <[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]
2025-11-20 00:36:25 +08:00
shiro.lee and GitHub
9b9bbb662b
fix: removing the Limit on the Number of Object Versions ( #819 ) ( #892 )
...
removing the Limit on the Number of Object Versions (#819 )
2025-11-19 22:34:26 +08:00
44f3f3d070
add standalone mode support ( #881 )
...
* add standalone mode support
* update readme file
* change non-root from 1000 to 10001
* delete self sign crt content
* modify security content
* fix synatx error for readme file.
Co-authored-by: Copilot <[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]
2025-11-19 13:56:21 +08:00
weisd and GitHub
a13ce08590
fix listbucket sign ( #890 )
2025-11-19 11:08:16 +08:00
55d44622ed
list object include deleted support ( #882 )
...
Co-authored-by: houseme <[email protected]
2025-11-18 21:51:10 +08:00
6da5766ea2
Implement adaptive buffer sizing with workload profiles - Production-Ready Implementation ( #878 )
...
* Initial plan
* Add workload profiles module with adaptive buffer sizing
Co-authored-by: houseme <[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]
2025-11-18 13:32:02 +08:00
weisd and GitHub
85bc0ce2d5
fix: filemeta version handling and delete operations ( #879 )
...
* fix filemeta version
* fix clippy
* fix delete version
* fix clippy/test
2025-11-18 09:24:22 +08:00
601f3456bc
Fix large file upload freeze with adaptive buffer sizing ( #869 )
...
* Initial plan
* Fix large file upload freeze by increasing StreamReader buffer size
Co-authored-by: houseme <[email protected] [email protected] [email protected] [email protected] [email protected] #871 )
* Initial plan
* Fix: Trigger self-heal when missing shards detected during read
- Added proactive heal detection in get_object_with_fileinfo
- When reading an object, now checks if any shards are missing even if read succeeds
- Sends low-priority heal request to reconstruct missing shards on rejoined nodes
- This fixes the issue where data written during node outage is not healed when node rejoins
Co-authored-by: houseme <[email protected] #873 )
* Initial plan
* Replace CRC libraries with unified crc-fast implementation
Co-authored-by: houseme <[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]
2025-11-17 23:15:20 +08:00
weisd and GitHub
1279baa72b
fix replication ( #875 )
2025-11-17 17:37:41 +08:00
weisd and GitHub
acdefb6703
fix read lock ( #866 )
2025-11-16 11:44:13 +08:00
b7964081ce
Fix KMS configuration synchronization across cluster nodes ( #855 )
...
* Initial plan
* Add KMS configuration persistence to cluster storage
Co-authored-by: houseme <[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]
2025-11-16 00:05:03 +08:00
Nugine and GitHub
f73fa59bf6
ci: fix dependabot ( #860 )
2025-11-15 22:35:59 +08:00
Nugine and GitHub
0b1b7832fe
ci: update s3s weekly ( #858 )
2025-11-15 22:05:03 +08:00
houseme and GitHub
c242957c6f
build(deps): bump the dependencies group with 8 updates ( #857 )
2025-11-15 19:51:07 +08:00
houseme and GitHub
55e3a1f7e0
fix(audit): prevent state transition when no targets exist ( #854 )
...
Avoid setting AuditSystemState::Starting when target list is empty.
Now checks target availability before state transition, keeping the
system in Stopped state if no enabled targets are found.
- Check targets.is_empty() before setting Starting state
- Return early with Ok(()) when no targets exist
- Maintain consistent state machine behavior
- Prevent transient "Starting" state with no actual targets
Resolves issue where audit system would incorrectly enter Starting
state even when configuration contained no enabled targets.
2025-11-14 20:23:21 +08:00
3cf565e847
delete sink file path env and update readme file with container user change ( #852 )
...
* update container user change in readme file
* delete sink file path env vars
---------
Co-authored-by: houseme <[email protected]
2025-11-14 13:00:15 +08:00
houseme and GitHub
9d553620cf
remove linux dep and upgrade Protocol Buffers and FlatBuffers ( #853 )
2025-11-14 12:50:55 +08:00
51584986e1
feat(obs): unify metrics initialization and fix exporter move error ( #851 )
...
* feat(obs): unify metrics initialization and fix exporter move error
- Fix Rust E0382 (use after move) by removing duplicate MetricExporter consumption.
- Consolidate MeterProvider construction into single Recorder builder path.
- Remove redundant Recorder::builder(...).install_global() call.
- Ensure PeriodicReader setup is performed only once (HTTP + optional stdout).
- Set global meter provider and metrics recorder exactly once.
- Preserve existing behavior for stdout/file vs HTTP modes.
- Minor cleanup: consistent resource reuse and interval handling.
* update telemetry.rs
Co-authored-by: Copilot <[email protected] [email protected]
2025-11-14 00:50:07 +08:00
majinghe and GitHub
93090adf7c
enhance security context part for k8s deployment ( #850 )
2025-11-13 18:18:19 +08:00
houseme and GitHub
d4817a4bea
fix: modify logger from warn to info ( #842 )
...
* fix: modify logger from warn to info
* upgrade version
2025-11-12 11:13:38 +08:00
houseme and GitHub
7e1a9e2ede
🔒 Upgrade Cryptography Libraries to Latest RC Versions ( #837 )
...
* fix
* chore: upgrade cryptography libraries to RC versions
- Upgrade aes-gcm to 0.11.0-rc.2 with rand_core support
- Upgrade chacha20poly1305 to 0.11.0-rc.2
- Upgrade argon2 to 0.6.0-rc.2 with std features
- Upgrade hmac to 0.13.0-rc.3
- Upgrade pbkdf2 to 0.13.0-rc.2
- Upgrade rsa to 0.10.0-rc.10
- Upgrade sha1 and sha2 to 0.11.0-rc.3
- Upgrade md-5 to 0.11.0-rc.3
These upgrades provide enhanced security features and performance
improvements while maintaining backward compatibility with existing
encryption workflows.
* add
* improve code
* fix
2025-11-11 21:10:03 +08:00
安正超 and GitHub
8a020ec4d9
wip ( #830 )
2025-11-11 09:34:58 +08:00
weisd and GitHub
77a3489ed2
fix list object err ( #831 )
...
fix list object err (#831 )
#827
#815
#635
#752
2025-11-10 23:42:15 +08:00
weisd and GitHub
5941062909
fix ( #828 )
2025-11-10 19:22:58 +08:00
houseme and GitHub
98be7df0f5
feat(storage): refactor audit and notification with OperationHelper ( #825 )
...
* improve code for audit
* improve code ecfs.rs
* improve code
* improve code for ecfs.rs
* feat(storage): refactor audit and notification with OperationHelper
This commit introduces a significant refactoring of the audit logging and event notification mechanisms within `ecfs.rs`.
The core of this change is the new `OperationHelper` struct, which encapsulates and simplifies the logic for both concerns. It replaces the previous `AuditHelper` and manual event dispatching.
Key improvements include:
- **Unified Handling**: `OperationHelper` manages both audit and notification builders, providing a single, consistent entry point for S3 operations.
- **RAII for Automation**: By leveraging the `Drop` trait, the helper automatically dispatches logs and notifications when it goes out of scope. This simplifies S3 method implementations and ensures cleanup even on early returns.
- **Fluent API**: A builder-like pattern with methods such as `.object()`, `.version_id()`, and `.suppress_event()` makes the code more readable and expressive.
- **Context-Aware Logic**: The helper's `.complete()` method intelligently populates log details based on the operation's `S3Result` and only triggers notifications on success.
- **Modular Design**: All helper logic is now isolated in `rustfs/src/storage/helper.rs`, improving separation of concerns and making `ecfs.rs` cleaner.
This refactoring significantly enhances code clarity, reduces boilerplate, and improves the robustness of logging and notification handling across the storage layer.
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* improve code for audit and notify
* fix
* fix
* fix
2025-11-10 17:30:50 +08:00
houseme and GitHub
b26aad4129
improve code for logger ( #822 )
...
* improve code for logger
* fix
2025-11-08 22:36:24 +08:00
5989589c3e
Update configuration.md ( #812 )
...
Escaping Pipe Character in the table "CLI Flags..."
Co-authored-by: loverustfs <[email protected]
2025-11-08 10:56:14 +08:00
majinghe and GitHub
4716454faa
add non root user support for container deployment ( #817 )
2025-11-08 10:00:14 +08:00
houseme and GitHub
29056a767a
Refactor Telemetry Initialization and Environment Utilities ( #811 )
...
* improve code for metrics
* improve code for metrics
* fix
* fix
* Refactor telemetry initialization and environment functions ordering
- Reorder functions in envs.rs by type size (8-bit to 64-bit, signed before unsigned) and add missing variants like get_env_opt_u16.
- Optimize init_telemetry to support three modes: stdout logging (default error level with span tracing), file rolling logs (size-based with retention), and HTTP-based observability with sub-endpoints (trace, metric, log) falling back to unified endpoint.
- Fix stdout logging issue by retaining WorkerGuard in OtelGuard to prevent premature release of async writer threads.
- Enhance observability mode with HTTP protocol, compression, and proper resource management.
- Update OtelGuard to include tracing_guard for stdout and flexi_logger_handles for file logging.
- Improve error handling and configuration extraction in OtelConfig.
* fix
* up
* fix
* fix
* improve code for obs
* fix
* fix
2025-11-07 20:01:54 +08:00
weisd and GitHub
e823922654
feat:add api error message ( #801 )
...
* feat:add api error message
* fix: check input
* fix: test
2025-11-07 09:53:49 +08:00
shiro.lee and GitHub
8203f9ff6f
fix: when the Object Lock configuration does not exist, an error message should be returned ( #771 ) ( #798 )
...
fix: when the Object Lock configuration does not exist, an error message should be returned (#771 ) (#798 )
2025-11-05 23:48:54 +08:00
houseme and GitHub
1b22a1e078
Refactor modify stdout ( #797 )
...
* fix
* fix
2025-11-05 20:04:28 +08:00
weisd and GitHub
461d5dff86
fix list max keys ( #795 )
2025-11-05 15:30:32 +08:00
houseme and GitHub
38f26b7c94
improve import，crate version，and copyright ( #790 )
2025-11-05 09:10:06 +08:00
eb7eb9c5a1
fix: resolve logic errors in ahm heal module ( #788 )
...
* fix: resolve logic errors in ahm heal module
- Fix response publishing logic in HealChannelProcessor to properly handle errors
- Fix negative index handling in DiskStatusChange event to fail fast instead of silently converting to 0
- Enhance timeout control in heal_erasure_set Step 3 loop to immediately respond to cancellation/timeout
- Add proper error propagation for task cancellation and timeout in bucket healing loop
* fix: stabilize performance impact measurement test
- Increase measurement count from 3 to 5 runs for better stability
- Increase workload from 5000 to 10000 operations for more accurate timing
- Use median of 5 measurements instead of single measurement
- Ensure with_scanner duration is at least baseline to avoid negative overhead
- Increase wait time for scanner state stabilization
* wip
* Update crates/ahm/src/heal/channel.rs
Co-authored-by: Copilot <[email protected] [email protected] [email protected]
2025-11-05 08:15:23 +08:00
houseme and GitHub
d934e3905b
Refactor telemetry initialization for non-production environments ( #789 )
...
* add dep `scopeguard`
* improve for tracing
* fix
* fix
* improve code for import
* add logger trace id
* fix
* fix
* fix
* fix
* fix
2025-11-05 00:55:08 +08:00
weisd and GitHub
6617372b33
fix rmdir versionid ( #784 )
2025-11-03 18:23:16 +08:00
weisd and GitHub
769778e565
fix iam ( #783 )
2025-11-03 17:39:51 +08:00
houseme and GitHub
a7f5c4af46
fix windows response ( #781 )
2025-11-03 12:49:39 +08:00
a9d5fbac54
build(deps): bump the dependencies group with 6 updates ( #777 )
...
* build(deps): bump the dependencies group with 6 updates
Bumps the dependencies group with 6 updates:
| Package | From | To |
| --- | --- | --- |
| [axum-extra](https://github.com/tokio-rs/axum ) | `0.10.3` | `0.12.0` |
| [aws-config](https://github.com/smithy-lang/smithy-rs ) | `1.8.8` | `1.8.10` |
| [aws-sdk-s3](https://github.com/awslabs/aws-sdk-rust ) | `1.109.0` | `1.110.0` |
| [aws-smithy-types](https://github.com/smithy-lang/smithy-rs ) | `1.3.3` | `1.3.4` |
| [clap](https://github.com/clap-rs/clap ) | `4.5.50` | `4.5.51` |
| [matchit](https://github.com/ibraheemdev/matchit ) | `0.8.4` | `0.9.0` |
Updates `axum-extra` from 0.10.3 to 0.12.0
- [Release notes](https://github.com/tokio-rs/axum/releases )
- [Changelog](https://github.com/tokio-rs/axum/blob/main/CHANGELOG.md )
- [Commits](https://github.com/tokio-rs/axum/compare/axum-extra-v0.10.3...axum-extra-v0.12.0 )
Updates `aws-config` from 1.8.8 to 1.8.10
- [Release notes](https://github.com/smithy-lang/smithy-rs/releases )
- [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/smithy-lang/smithy-rs/commits )
Updates `aws-sdk-s3` from 1.109.0 to 1.110.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases )
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits )
Updates `aws-smithy-types` from 1.3.3 to 1.3.4
- [Release notes](https://github.com/smithy-lang/smithy-rs/releases )
- [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md )
- [Commits](https://github.com/smithy-lang/smithy-rs/commits )
Updates `clap` from 4.5.50 to 4.5.51
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/clap_complete-v4.5.50...clap_complete-v4.5.51 )
Updates `matchit` from 0.8.4 to 0.9.0
- [Release notes](https://github.com/ibraheemdev/matchit/releases )
- [Commits](https://github.com/ibraheemdev/matchit/compare/v0.8.4...v0.9.0 )
---
updated-dependencies:
- dependency-name: axum-extra
dependency-version: 0.12.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: dependencies
- dependency-name: aws-config
dependency-version: 1.8.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
- dependency-name: aws-sdk-s3
dependency-version: 1.110.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: dependencies
- dependency-name: aws-smithy-types
dependency-version: 1.3.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
- dependency-name: clap
dependency-version: 4.5.51
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
- dependency-name: matchit
dependency-version: 0.9.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: dependencies
...
Signed-off-by: dependabot[bot] <[email protected] [email protected] [email protected]
2025-11-03 00:34:54 +08:00
