Commit Graph
45 Commits
Author SHA1 Message Date
e12d63234a test(targets): cover Kafka check SASL validation (#3136)
test(targets): cover kafka check sasl validation

Co-authored-by: houseme <[email protected]>
2026-05-31 04:47:21 +00:00
5bdbd66d09 feat(targets): support Kafka SASL auth (#3128)
* feat(targets): support Kafka SASL auth

* fix(targets): infer Kafka SASL when enable is blank

---------

Co-authored-by: Henry Guo <[email protected]>
2026-05-30 11:01:17 +00:00
ac97ceb744 fix(config): restore default credential startup (#3114)
* fix(config): restore default credential startup

* fix: align e2e credentials with server env

* fix(config): restore default credential consistency

---------

Co-authored-by: houseme <[email protected]>
2026-05-29 11:52:46 +00:00
housemeandGitHub d74e6eb042 refactor(tls): centralize runtime foundation (#3065)
* refactor(targets): move notify net helpers from utils

* refactor(tls): centralize runtime foundation

* refactor(targets): move notify net helpers from utils

* refactor(tls): centralize runtime foundation

* feat(tls-runtime): add TLS debug state and admin handler

* refactor(tls-runtime): unify TLS debug consumer status view

* fix(tls): address PR3065 review feedback

* refactor(tls): align debug status payload types

* refactor(targets): harden TLS hot reload paths

* fix(targets): resolve review-4348251652 findings

* fix(targets): finalize tls runtime review follow-ups

* fix(targets): harden tls reload and review follow-ups

* fix(targets): align tls reload handling across targets

* fix(targets): finalize tls reload state and metrics updates

* chore(deps): trim unused TLS deps

* style(targets): normalize TLS reload formatting

* refactor(targets): introduce tls runtime adapter path

* chore: update workspace manifests for tls refactor

* fix(tls): stabilize material reload and audit workflow

* fix(targets): refresh tls fingerprint flow across sinks

* fix(tls): align runtime coordinator and http reader updates

* fix(sftp): simplify protocol error mapping

* fix(tls): harmonize material loading behavior

* fix(server): finalize tls material wiring in startup flow

* fix(protos): tighten tls generation cache and deps
2026-05-24 06:41:15 +00:00
housemeandGitHub dcbffb084f chore(deps): refresh workspace deps and linux fs_type gating (#3030)
* chore(deps): refresh workspace deps and linux fs_type gating

- refresh workspace dependency pins and lockfile updates

- remove now-unused crate dependency entries in multiple Cargo.toml files

- enable profiling export defaults in config and scripts/run.sh

- gate os::fs_type module/function/tests to Linux to avoid non-Linux dead_code warnings

* fix(utils): simplify fs_type linux gating

- keep fs_type module-level linux cfg in os::mod

- remove redundant linux cfg on get_fs_type and test module

* chore(deps): bump s3s git revision

- update workspace s3s dependency to rev 507e1312b211c3ddc214b03875d6fabd15d22ed5

- refresh Cargo.lock source entry for s3s

* chore(dev): allow mysql_async git source and env overrides

- allow mysql_async git source in deny.toml allow-git list

- make scripts/run.sh core env vars overrideable via existing shell env

* fix(utils): import get_fs_type in fs_type tests

- add explicit super::get_fs_type import in fs_type test module

- fix Linux E0425 unresolved function errors in unit tests

* chore(dev): tune run script observability defaults

- make profiling export env overrideable in scripts/run.sh

- set RUSTFS_OBS_SAMPLE_RATIO default from 2.0 to 1.0

- update allow-git review window comments in deny.toml

* test(obs): stabilize profiling env alias tests
2026-05-20 15:03:07 +00:00
housemeandGitHub 73bde843d6 refactor(s3): consolidate semantic boundaries and remove s3-common (#3012)
* refactor(common): introduce rustfs-data-usage core crate

* refactor(concurrency): migrate workers crate into concurrency

* refactor(crypto): migrate appauth token APIs into crypto

* fix docs urls

* remove unused crate

* refactor(data-usage): switch consumers to rustfs-data-usage

* chore(fmt): apply cargo fmt and lockfile sync

* refactor(common): remove data_usage compatibility re-export

* refactor(capacity): move capacity_scope to object-capacity

* refactor(io-metrics): relocate internode metrics from common

* refactor(common): decouple scanner report from madmin

* chore(fmt): normalize import ordering after pre-commit

* refactor(s3): split s3 types and ops crates

* refactor(s3): centralize event version and safe parsing

* refactor(s3): add op-event compatibility guardrails

* refactor(s3): add runtime op-event mismatch observability

* refactor(s3): extract delete event mapping helper

* refactor(s3): extract put event mapping helper

* refactor(s3): consolidate remaining event semantic helpers

* refactor(s3): add op-event coverage checks and observability alerts

* refactor(s3-ops): consolidate op-event semantic mapping

* refactor(scanner): remove last_minute wrapper module

* refactor(scanner): consolidate duplicated data usage models
2026-05-19 12:50:25 +00:00
bca8b08c2b fix: handle Windows paths in pre-commit tests (#2974)
Co-authored-by: houseme <[email protected]>
2026-05-15 14:04:51 +00:00
81754d80b3 refactor(targets): unify queue/connectivity handling and coverage (#2953)
Co-authored-by: Claude Opus 4.7 <[email protected]>
Co-authored-by: marshawcoco <[email protected]>
2026-05-14 04:31:23 +00:00
安正超andGitHub 404fc816e9 test(targets): cover MySQL probe validation (#2907) 2026-05-11 01:55:48 +00:00
0375dd39bb fix(targets): handle MySQL DSN scheme case (#2903)
Co-authored-by: houseme <[email protected]>
2026-05-10 14:25:22 +00:00
bafff6d207 feat(targets): add check_mysql_server_available probe function (#2884)
Signed-off-by: JaySon-Huang <[email protected]>
Co-authored-by: houseme <[email protected]>
2026-05-10 11:53:45 +00:00
cb54cecf8c docs(targets): sync AGENTS.md and test doc comments with code (#2881)
Signed-off-by: houseme <[email protected]>
Co-authored-by: houseme <[email protected]>
2026-05-09 03:06:11 +00:00
安正超andGitHub 9288f1ed3a fix(targets): handle postgres dsn redaction scheme case (#2886) 2026-05-09 01:58:23 +00:00
81ad48dac2 feat(targets): add AMQP support for notify and audit (#2879)
Co-authored-by: Hyesook Yun <[email protected]>
2026-05-09 01:56:26 +00:00
安正超andGitHub 9df57b8cd8 test(targets): cover Redis env config loading (#2857) 2026-05-07 13:08:11 +00:00
housemeandGitHub fd37a7d01e fix(targets): probe webhook health by host port (#2854) 2026-05-07 11:52:45 +00:00
5431b9273d feat(targets): complete redis mysql postgres target wiring (#2842)
Signed-off-by: jaehanbyun <[email protected]>
Signed-off-by: houseme <[email protected]>
Signed-off-by: Gunther Xing <[email protected]>
Signed-off-by: JaySon-Huang <[email protected]>
Co-authored-by: jaehanbyun <[email protected]>
Co-authored-by: Gunther Xing <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: cxymds <[email protected]>
Co-authored-by: JaySon <[email protected]>
Co-authored-by: 安正超 <[email protected]>
2026-05-07 10:00:59 +00:00
368ef0f16c refactor(targets): unify endpoint source/merge logic and bump rustfs-kafka-async to v1.2.0 (#2654)
Co-authored-by: Filipe Monteiro <[email protected]>
Co-authored-by: cxymds <[email protected]>
Co-authored-by: weisd <[email protected]>
Co-authored-by: loverustfs <[email protected]>
2026-04-23 09:14:36 +00:00
安正超andGitHub 8add0126f5 test(targets): cover target config redaction (#2638) 2026-04-22 02:51:41 +00:00
housemeandGitHub 3ac1d2ab0b fix(security): redact target debug logs and remove eval-based bench hook (#2637) 2026-04-21 21:21:01 +00:00
安正超andGitHub 1525143a04 test: cover NATS and Pulsar config validation (#2623) 2026-04-21 01:59:05 +00:00
housemeandGitHub b7a945e453 fix(madmin): restore server_info msgpack compatibility across mixed nodes (#2621) 2026-04-21 01:09:25 +00:00
housemeandGitHub 3796b684f0 feat(targets): add NATS and Pulsar target support (#2618) 2026-04-21 00:28:19 +00:00
housemeandGitHub 1cbf156559 refactor(obs): migrate metrics runtime/schema and tighten migration guards (#2584) 2026-04-18 07:51:15 +00:00
TungliesandGitHub 49366ee200 chore(lint): clippy rules redundant_clone (#2554) 2026-04-15 13:54:07 +00:00
TungliesandGitHub f57dd5a3c7 chore(lint): clippy rules needless_collect (#2522) 2026-04-15 10:00:03 +00:00
979626c370 refactor(utils): decouple config deps and move sys helpers (#2520)
Co-authored-by: houseme <[email protected]>
2026-04-13 21:05:03 +08:00
d6158c0481 feat(mqtt): migrate client and harden TLS config (#2413)
Signed-off-by: houseme <[email protected]>
Co-authored-by: heihutu <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: houseme <[email protected]>
Co-authored-by: 安正超 <[email protected]>
Co-authored-by: Copilot <[email protected]>
2026-04-07 15:28:29 +08:00
d2901fd78c feat(admin): add audit target APIs and harden target source handling (#2350)
Signed-off-by: houseme <[email protected]>
Co-authored-by: 安正超 <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: houseme <[email protected]>
2026-04-04 09:07:22 +08:00
9ce3c7742c fix(targets): pass credentials to MQTT broker availability check (#2192)
Signed-off-by: houseme <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
2026-03-17 23:53:28 +08:00
housemeandGitHub 60aa47bf61 feat(storage): integrate S3Operation into OperationHelper for unified metrics and audit (#2103) 2026-03-08 17:57:33 +08:00
d17d2083d4 feat(targets): enhance webhook TLS support with custom CA and skip-verify (#1994)
Co-authored-by: Copilot <[email protected]>
Co-authored-by: houseme <[email protected]>
Co-authored-by: heihutu <[email protected]>
2026-02-27 21:24:49 +08:00
548a39ffe7 fix: return error instead of silently ignoring invalid ARNs in notification config (#1528)
Co-authored-by: Claude Opus 4.5 <[email protected]>
2026-01-16 16:12:55 +08:00
housemeandGitHub a95e549430 Fix/fix improve for audit (#1418) 2026-01-07 18:05:52 +08:00
housemeandGitHub 8d7cd4cb1b chore: upgrade dependencies and migrate to aws-lc-rs (#1333) 2026-01-02 00:02:34 +08:00
8dd3e8b534 fix: decode form-urlencoded object names in webhook/mqtt Key field (#1210)
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: houseme <[email protected]>
Co-authored-by: houseme <[email protected]>
2025-12-20 01:31:09 +08:00
4abfc9f554 Fix/fix event 1216 (#1191)
Signed-off-by: loverustfs <[email protected]>
Co-authored-by: loverustfs <[email protected]>
2025-12-19 12:07:07 +08:00
mythrnrandGitHub 94d5b1c1e4 fix: format of bucket event notifications (#1138) 2025-12-16 20:44:57 +08:00
b7964081ce Fix KMS configuration synchronization across cluster nodes (#855)
* Initial plan

* Add KMS configuration persistence to cluster storage

Co-authored-by: houseme <[email protected]>

* Apply code formatting to KMS configuration changes

Co-authored-by: houseme <[email protected]>

* add comment

* fix fmt

* fix

* Fix overlapping dependabot cargo configurations

Co-authored-by: houseme <[email protected]>

* improve code for comment and replace  `Once_Cell` to `std::sync::OnceLock`

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: houseme <[email protected]>
Co-authored-by: houseme <[email protected]>
Co-authored-by: loverustfs <[email protected]>
2025-11-16 00:05:03 +08:00
aac9b1edb7 chore: improve event and docker-compose ,Improve the permissions of the endpoint health interface, upgrade otel from 0.30.0 to 0.31.0 (#620)
* feat: improve code for notify

* upgrade starshard version

* upgrade version

* Fix ETag format to comply with HTTP standards by wrapping with quotes (#592)

* Initial plan

* Fix ETag format to comply with HTTP standards by wrapping with quotes

Co-authored-by: overtrue <[email protected]>

* bufigx

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: overtrue <[email protected]>
Co-authored-by: overtrue <[email protected]>

* Improve lock (#596)

* improve lock

Signed-off-by: Mu junxiang <[email protected]>

* feat(tests): add wait_for_object_absence helper and improve lifecycle test reliability

Signed-off-by: Mu junxiang <[email protected]>

* chore: remove dirty docs

Signed-off-by: Mu junxiang <[email protected]>

---------

Signed-off-by: Mu junxiang <[email protected]>

* feat(append): implement object append operations with state tracking (#599)

* feat(append): implement object append operations with state tracking

Signed-off-by: junxiang Mu <[email protected]>

* chore: rebase

Signed-off-by: junxiang Mu <[email protected]>

---------

Signed-off-by: junxiang Mu <[email protected]>

* build(deps): upgrade s3s (#595)

Co-authored-by: loverustfs <[email protected]>

* fix: validate mqtt broker

* improve code for `import`

* upgrade otel relation crates version

* fix:dep("jsonwebtoken") feature = 'rust_crypto'

* fix

* fix

* fix

* upgrade version

* improve code for ecfs

* chore: improve event and docker-compose ,Improve the permissions of the `endpoint` health interface

* fix

* fix

* fix

* fix

* improve code

* fix

---------

Signed-off-by: Mu junxiang <[email protected]>
Signed-off-by: junxiang Mu <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: overtrue <[email protected]>
Co-authored-by: overtrue <[email protected]>
Co-authored-by: guojidan <[email protected]>
Co-authored-by: Nugine <[email protected]>
Co-authored-by: loverustfs <[email protected]>
2025-10-11 09:08:25 +08:00
housemeandGitHub 9b7f4d477a Fix Tokio Runtime Initialization: Remove Private API Usage and Ensure IO Enabled (#587)
* fix: remove code

* improve code for tokio runtime config

* improve code for main

* fix: add tokio enable_all

* upgrade version

* improve for Cargo.toml
2025-09-24 22:23:31 +08:00
29b0935be7 RustFS rustfs-audit Complete Implementation with Enterprise Observability (#557)
* Initial plan

* Implement core audit system with multi-target fan-out and configuration management

Co-authored-by: houseme <[email protected]>

* Changes before error encountered

Co-authored-by: houseme <[email protected]>

* Complete audit system with comprehensive observability and test coverage

Co-authored-by: houseme <[email protected]>

* improve code

* fix

* improve code

* fix test

* fix test

* fix

* add `rustfs-audit` to `rustfs`

* upgrade crate version

* fmt

* fmt

* fix

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: houseme <[email protected]>
Co-authored-by: houseme <[email protected]>
2025-09-24 08:23:46 +08:00
29c004d935 feat: enhance console separation with enterprise-grade security, monitoring, and advanced tower-http integration (#513)
* Initial plan

* feat: implement console service separation from endpoint

Co-authored-by: houseme <[email protected]>

* feat: add console separation documentation and tests

Co-authored-by: houseme <[email protected]>

* feat: enhance console separation with configurable CORS and improved Docker support

Co-authored-by: houseme <[email protected]>

* feat: implement enhanced console separation with security hardening and monitoring

Co-authored-by: houseme <[email protected]>

* refactor: implement console TLS following endpoint logic and improve configuration

Co-authored-by: houseme <[email protected]>

* add tower-http feature "timeout|limit"

* add dependencies crates `axum-server`

* refactor: reconstruct console server with enhanced tower-http features and environment variables

Co-authored-by: houseme <[email protected]>

* upgrade dep

* improve code for dns and console port `:9001`

* improve code

* fix

* docs: comprehensive improvement of console separation documentation and Docker deployment standards

Co-authored-by: houseme <[email protected]>

* fmt

* add logs

* improve code for Config handler

* remove logs

* fix

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: houseme <[email protected]>
Co-authored-by: houseme <[email protected]>
2025-09-13 14:48:14 +08:00
housemeandGitHub 4e00110bfe add bucket notification configuration (#502) 2025-09-10 00:56:27 +08:00
+4 adc07e5209 feat(targets): extract targets module into a standalone crate (#441)
* init audit logger module

* add audit webhook default config kvs

* feat: Add comprehensive tests for authentication module (#309)

* feat: add comprehensive tests for authentication module

- Add 33 unit tests covering all public functions in auth.rs
- Test IAMAuth struct creation and secret key validation
- Test check_claims_from_token with various credential types and scenarios
- Test session token extraction from headers and query parameters
- Test condition values generation for different user types
- Test query parameter parsing with edge cases
- Test Credentials helper methods (is_expired, is_temp, is_service_account)
- Ensure tests handle global state dependencies gracefully
- All tests pass successfully with 100% coverage of testable functions

* style: fix code formatting issues

* Add verification script for checking PR branch statuses and tests

Co-authored-by: anzhengchao <[email protected]>

* fix: resolve clippy uninlined format args warning

---------

Co-authored-by: Cursor Agent <[email protected]>

* feat: add basic tests for core storage module (#313)

* feat: add basic tests for core storage module

- Add 6 unit tests for FS struct and basic functionality
- Test FS creation, Debug and Clone trait implementations
- Test RUSTFS_OWNER constant definition and values
- Test S3 error code creation and handling
- Test compression format detection for common file types
- Include comprehensive documentation about integration test needs

Note: Full S3 API testing requires complex setup with storage backend,
global configuration, and network infrastructure - better suited for
integration tests rather than unit tests.

* style: fix code formatting issues

* fix: resolve clippy warnings in storage tests

---------

Co-authored-by: Cursor Agent <[email protected]>

* feat: add tests for admin handlers module (#314)

* feat: add tests for admin handlers module

- Add 5 new unit tests for admin handler functionality
- Test AccountInfo struct creation, serialization and default values
- Test creation of all admin handler structs (13 handlers)
- Test HealOpts JSON serialization and deserialization
- Test HealOpts URL encoding/decoding with proper field types
- Maintain existing test while adding comprehensive coverage
- Include documentation about integration test requirements

All tests pass successfully with proper error handling for complex dependencies.

* style: fix code formatting issues

* fix: resolve clippy warnings in admin handlers tests

---------

Co-authored-by: Cursor Agent <[email protected]>

* build(deps): bump the dependencies group with 3 updates (#326)

* perf: avoid transmitting parity shards when the object is good (#322)

* upgrade version

* Fix: fix data integrity check

Signed-off-by: junxiang Mu <[email protected]>

* Fix: Separate Clippy's fix and check commands into two commands.

Signed-off-by: junxiang Mu <[email protected]>

* fix: miss inline metadata (#345)

* Update dependabot.yml

* fix: Fixed an issue where the list_objects_v2 API did not return dire… (#352)

* fix: Fixed an issue where the list_objects_v2 API did not return directory names when they conflicted with file names in the same bucket (e.g., test/ vs. test.txt, aaa/ vs. aaa.csv) (#335)

* fix: adjusted the order of directory listings

* init

* fix

* fix

* feat: add docker usage for rustfs mcp (#365)

* feat: enhance metadata extraction with object name for MIME type detection

Signed-off-by: junxiang Mu <[email protected]>

* Feature: lock support auto release

Signed-off-by: junxiang Mu <[email protected]>

* improve lock

Signed-off-by: junxiang Mu <[email protected]>

* Fix: fix scanner detect

Signed-off-by: junxiang Mu <[email protected]>

* Fix: clippy && fmt

Signed-off-by: junxiang Mu <[email protected]>

* refactor(ecstore): Optimize memory usage for object integrity verification

Change the object integrity verification from reading all data to streaming processing to avoid memory overflow caused by large objects.

Modify the TLS key log check to use environment variables directly instead of configuration constants.

Add memory limits for object data reading in the AHM module.

Signed-off-by: junxiang Mu <[email protected]>

* Chore: reduce PR template checklist

Signed-off-by: junxiang Mu <[email protected]>

* Chore: remove comment code (#376)

Signed-off-by: junxiang Mu <[email protected]>

* chore: upgrade actions/checkout from v4 to v5 (#381)

* chore: upgrade actions/checkout from v4 to v5

- Update GitHub Actions checkout action version
- Ensure compatibility with latest workflow features
- Maintain existing checkout behavior and configuration

* upgrade version

* fix

* add and improve code for notify

* feat: extend rustfs mcp with bucket creation and deletion (#416)

* feat: extend rustfs mcp with bucket creation and deletion

* update file to fix pipeline error

* change variable name to fix pipeline error

* fix(ecstore): add async-recursion to resolve nightly trait solver reg… (#415)

* fix(ecstore): add async-recursion to resolve nightly trait solver regression

The newest nightly compiler switched to the new trait solver, which
currently rejects async recursive functions that were previously accepted.
This causes the following compilation failures:

- `LocalDisk::delete_file()`
- `LocalDisk::scan_dir()`

Add `async-recursion` as a workspace dependency and annotate both functions with `#[async_recursion]` so that the crate compiles cleanly with the latest nightly and will continue to build once the new solver lands in stable.

Signed-off-by: reigadegr <[email protected]>

* fix: resolve duplicate bound error in scan_dir function

Replaced inline trait bounds with where clause to avoid duplication caused by macro expansion.

Signed-off-by: reigadegr <[email protected]>

---------

Signed-off-by: reigadegr <[email protected]>
Co-authored-by: 安正超 <[email protected]>

* fix:make bucket exists (#428)

* feat: include user-defined metadata in S3 response (#431)

* fix: simplify Docker entrypoint following efficient user switching pattern (#421)

* fix: simplify Docker entrypoint following efficient user switching pattern

- Remove ALL file permission modifications (no chown at all)
- Use chroot --userspec or gosu to switch user context
- Extremely simple and fast implementation
- Zero filesystem modifications for permissions

Fixes #388

* Update entrypoint.sh

Co-authored-by: Copilot <[email protected]>

* Update entrypoint.sh

Co-authored-by: Copilot <[email protected]>

* Update entrypoint.sh

Co-authored-by: Copilot <[email protected]>

* wip

* wip

* wip

---------

Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Copilot <[email protected]>

* docs: update doc/docker-data-dir README.md (#432)

* add targets crates

* feat(targets): extract targets module into a standalone crate

- Move all target-related code (MQTT, Webhook, etc.) into a new `targets` crate
- Update imports and dependencies to reference the new crate
- Refactor interfaces to ensure compatibility with the new crate structure
- Adjust Cargo.toml and workspace configuration accordingly

* fix

* fix

---------

Signed-off-by: junxiang Mu <[email protected]>
Signed-off-by: reigadegr <[email protected]>
Co-authored-by: 安正超 <[email protected]>
Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: zzhpro <[email protected]>
Co-authored-by: junxiang Mu <[email protected]>
Co-authored-by: weisd <[email protected]>
Co-authored-by: shiro.lee <[email protected]>
Co-authored-by: majinghe <[email protected]>
Co-authored-by: guojidan <[email protected]>
Co-authored-by: reigadegr <[email protected]>
Co-authored-by: 0xdx2 <[email protected]>
Co-authored-by: Copilot <[email protected]>
2025-08-21 22:33:07 +08:00