Commit Graph
37 Commits
Author SHA1 Message Date
036741cb1c refactor(request-id): align contracts and lock field names (#3454)
* refactor(request-id): align header log and trace contracts

* test(contract): lock external request-id field names

* chore(deps): drop unused rustfs-ecstore links

Co-Authored-By: heihutu <[email protected]>
2026-06-15 01:59:11 +08:00
housemeandGitHub e8012bd1ba refactor(logging): normalize admin telemetry and error messages (#3430) 2026-06-14 13:27:10 +08:00
housemeandGitHub 82af181dcf refactor(logging): unify governance runtime events (#3367) 2026-06-11 22:26:02 +08:00
housemeandGitHub 0a987d870b refactor(logging): reduce runtime noise (#3363) 2026-06-11 19:49:01 +08:00
安正超andGitHub ca58d7f0ec refactor(config): migrate server config consumers (#3353) 2026-06-11 17:04:50 +08:00
housemeandGitHub dcbffb084f chore(deps): refresh workspace deps and linux fs_type gating (#3030)
* chore(deps): refresh workspace deps and linux fs_type gating

- refresh workspace dependency pins and lockfile updates

- remove now-unused crate dependency entries in multiple Cargo.toml files

- enable profiling export defaults in config and scripts/run.sh

- gate os::fs_type module/function/tests to Linux to avoid non-Linux dead_code warnings

* fix(utils): simplify fs_type linux gating

- keep fs_type module-level linux cfg in os::mod

- remove redundant linux cfg on get_fs_type and test module

* chore(deps): bump s3s git revision

- update workspace s3s dependency to rev 507e1312b211c3ddc214b03875d6fabd15d22ed5

- refresh Cargo.lock source entry for s3s

* chore(dev): allow mysql_async git source and env overrides

- allow mysql_async git source in deny.toml allow-git list

- make scripts/run.sh core env vars overrideable via existing shell env

* fix(utils): import get_fs_type in fs_type tests

- add explicit super::get_fs_type import in fs_type test module

- fix Linux E0425 unresolved function errors in unit tests

* chore(dev): tune run script observability defaults

- make profiling export env overrideable in scripts/run.sh

- set RUSTFS_OBS_SAMPLE_RATIO default from 2.0 to 1.0

- update allow-git review window comments in deny.toml

* test(obs): stabilize profiling env alias tests
2026-05-20 15:03:07 +00:00
housemeandGitHub 73bde843d6 refactor(s3): consolidate semantic boundaries and remove s3-common (#3012)
* refactor(common): introduce rustfs-data-usage core crate

* refactor(concurrency): migrate workers crate into concurrency

* refactor(crypto): migrate appauth token APIs into crypto

* fix docs urls

* remove unused crate

* refactor(data-usage): switch consumers to rustfs-data-usage

* chore(fmt): apply cargo fmt and lockfile sync

* refactor(common): remove data_usage compatibility re-export

* refactor(capacity): move capacity_scope to object-capacity

* refactor(io-metrics): relocate internode metrics from common

* refactor(common): decouple scanner report from madmin

* chore(fmt): normalize import ordering after pre-commit

* refactor(s3): split s3 types and ops crates

* refactor(s3): centralize event version and safe parsing

* refactor(s3): add op-event compatibility guardrails

* refactor(s3): add runtime op-event mismatch observability

* refactor(s3): extract delete event mapping helper

* refactor(s3): extract put event mapping helper

* refactor(s3): consolidate remaining event semantic helpers

* refactor(s3): add op-event coverage checks and observability alerts

* refactor(s3-ops): consolidate op-event semantic mapping

* refactor(scanner): remove last_minute wrapper module

* refactor(scanner): consolidate duplicated data usage models
2026-05-19 12:50:25 +00:00
81754d80b3 refactor(targets): unify queue/connectivity handling and coverage (#2953)
Co-authored-by: Claude Opus 4.7 <[email protected]>
Co-authored-by: marshawcoco <[email protected]>
2026-05-14 04:31:23 +00:00
bafff6d207 feat(targets): add check_mysql_server_available probe function (#2884)
Signed-off-by: JaySon-Huang <[email protected]>
Co-authored-by: houseme <[email protected]>
2026-05-10 11:53:45 +00:00
housemeandGitHub 53ec1b95d8 keep sftp e2e tests buildable (#2897) 2026-05-10 11:52:15 +00:00
81ad48dac2 feat(targets): add AMQP support for notify and audit (#2879)
Co-authored-by: Hyesook Yun <[email protected]>
2026-05-09 01:56:26 +00:00
5431b9273d feat(targets): complete redis mysql postgres target wiring (#2842)
Signed-off-by: jaehanbyun <[email protected]>
Signed-off-by: houseme <[email protected]>
Signed-off-by: Gunther Xing <[email protected]>
Signed-off-by: JaySon-Huang <[email protected]>
Co-authored-by: jaehanbyun <[email protected]>
Co-authored-by: Gunther Xing <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: cxymds <[email protected]>
Co-authored-by: JaySon <[email protected]>
Co-authored-by: 安正超 <[email protected]>
2026-05-07 10:00:59 +00:00
housemeandGitHub b96ccfd110 build(deps): bump the dependencies group with 7 updates (#2676) 2026-04-25 02:15:14 +00:00
2705e3f53b fix(audit): remove unwrap from target shutdown path (#2668)
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
2026-04-24 09:25:11 +00:00
368ef0f16c refactor(targets): unify endpoint source/merge logic and bump rustfs-kafka-async to v1.2.0 (#2654)
Co-authored-by: Filipe Monteiro <[email protected]>
Co-authored-by: cxymds <[email protected]>
Co-authored-by: weisd <[email protected]>
Co-authored-by: loverustfs <[email protected]>
2026-04-23 09:14:36 +00:00
housemeandGitHub 3796b684f0 feat(targets): add NATS and Pulsar target support (#2618) 2026-04-21 00:28:19 +00:00
housemeandGitHub 1cbf156559 refactor(obs): migrate metrics runtime/schema and tighten migration guards (#2584) 2026-04-18 07:51:15 +00:00
579b124726 lint: clippy rules or_fun_call (#2561)
Co-authored-by: houseme <[email protected]>
2026-04-16 02:48:39 +00:00
d6158c0481 feat(mqtt): migrate client and harden TLS config (#2413)
Signed-off-by: houseme <[email protected]>
Co-authored-by: heihutu <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: houseme <[email protected]>
Co-authored-by: 安正超 <[email protected]>
Co-authored-by: Copilot <[email protected]>
2026-04-07 15:28:29 +08:00
d2901fd78c feat(admin): add audit target APIs and harden target source handling (#2350)
Signed-off-by: houseme <[email protected]>
Co-authored-by: 安正超 <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: houseme <[email protected]>
2026-04-04 09:07:22 +08:00
05dc131a49 perf(storage): optimize internode RPC transfer path (#2262)
Co-authored-by: momoda693 <[email protected]>
2026-03-23 17:09:49 +08:00
3c28f0a0ba feat(metrics): migrate system monitoring from rustfs-obs to rustfs-metrics (#2242)
Co-authored-by: houseme <[email protected]>
2026-03-20 18:52:33 +08:00
housemeandGitHub 60aa47bf61 feat(storage): integrate S3Operation into OperationHelper for unified metrics and audit (#2103) 2026-03-08 17:57:33 +08:00
d17d2083d4 feat(targets): enhance webhook TLS support with custom CA and skip-verify (#1994)
Co-authored-by: Copilot <[email protected]>
Co-authored-by: houseme <[email protected]>
Co-authored-by: heihutu <[email protected]>
2026-02-27 21:24:49 +08:00
housemeandGitHub a95e549430 Fix/fix improve for audit (#1418) 2026-01-07 18:05:52 +08:00
housemeandGitHub 8d7cd4cb1b chore: upgrade dependencies and migrate to aws-lc-rs (#1333) 2026-01-02 00:02:34 +08:00
housemeandGitHub 08f1a31f3f Fix notification event stream cleanup, add bounded send concurrency, and reduce overhead (#1224) 2025-12-22 00:57:05 +08:00
4abfc9f554 Fix/fix event 1216 (#1191)
Signed-off-by: loverustfs <[email protected]>
Co-authored-by: loverustfs <[email protected]>
2025-12-19 12:07:07 +08:00
housemeandGitHub 55e3a1f7e0 fix(audit): prevent state transition when no targets exist (#854)
Avoid setting AuditSystemState::Starting when target list is empty.
Now checks target availability before state transition, keeping the
system in Stopped state if no enabled targets are found.

- Check targets.is_empty() before setting Starting state
- Return early with Ok(()) when no targets exist
- Maintain consistent state machine behavior
- Prevent transient "Starting" state with no actual targets

Resolves issue where audit system would incorrectly enter Starting
state even when configuration contained no enabled targets.
2025-11-14 20:23:21 +08:00
housemeandGitHub 98be7df0f5 feat(storage): refactor audit and notification with OperationHelper (#825)
* improve code for audit

* improve code ecfs.rs

* improve code

* improve code for ecfs.rs

* feat(storage): refactor audit and notification with OperationHelper

This commit introduces a significant refactoring of the audit logging and event notification mechanisms within `ecfs.rs`.

The core of this change is the new `OperationHelper` struct, which encapsulates and simplifies the logic for both concerns. It replaces the previous `AuditHelper` and manual event dispatching.

Key improvements include:

- **Unified Handling**: `OperationHelper` manages both audit and notification builders, providing a single, consistent entry point for S3 operations.
- **RAII for Automation**: By leveraging the `Drop` trait, the helper automatically dispatches logs and notifications when it goes out of scope. This simplifies S3 method implementations and ensures cleanup even on early returns.
- **Fluent API**: A builder-like pattern with methods such as `.object()`, `.version_id()`, and `.suppress_event()` makes the code more readable and expressive.
- **Context-Aware Logic**: The helper's `.complete()` method intelligently populates log details based on the operation's `S3Result` and only triggers notifications on success.
- **Modular Design**: All helper logic is now isolated in `rustfs/src/storage/helper.rs`, improving separation of concerns and making `ecfs.rs` cleaner.

This refactoring significantly enhances code clarity, reduces boilerplate, and improves the robustness of logging and notification handling across the storage layer.

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* improve code for audit and notify

* fix

* fix

* fix
2025-11-10 17:30:50 +08:00
housemeandGitHub 29056a767a Refactor Telemetry Initialization and Environment Utilities (#811)
* improve code for metrics

* improve code for metrics

* fix

* fix

* Refactor telemetry initialization and environment functions ordering

- Reorder functions in envs.rs by type size (8-bit to 64-bit, signed before unsigned) and add missing variants like get_env_opt_u16.
- Optimize init_telemetry to support three modes: stdout logging (default error level with span tracing), file rolling logs (size-based with retention), and HTTP-based observability with sub-endpoints (trace, metric, log) falling back to unified endpoint.
- Fix stdout logging issue by retaining WorkerGuard in OtelGuard to prevent premature release of async writer threads.
- Enhance observability mode with HTTP protocol, compression, and proper resource management.
- Update OtelGuard to include tracing_guard for stdout and flexi_logger_handles for file logging.
- Improve error handling and configuration extraction in OtelConfig.

* fix

* up

* fix

* fix

* improve code for obs

* fix

* fix
2025-11-07 20:01:54 +08:00
安正超andGitHub dd47fcf2a8 fix: restore localized samples in tests (#749)
* fix: restore required localized examples

* style: fix formatting issues
2025-10-29 13:16:31 +08:00
cd1e244c68 Refactor: Introduce content checksums and improve multipart/object metadata handling (#671)
* feat:  adapt to s3s typed etag support

* refactor: move replication struct to rustfs_filemeta, fix filemeta transition bug

* add head_object checksum, filter object metadata output

* fix multipart checksum

* fix multipart checksum

* add content md5,sha256 check

* fix test

* fix cargo

---------

Co-authored-by: overtrue <[email protected]>
2025-10-20 23:46:13 +08:00
42d3645d6f fix(targets): make target removal and reload transactional; prevent reappearing entries (#662)
* feat: improve code for notify

* upgrade starshard version

* upgrade version

* Fix ETag format to comply with HTTP standards by wrapping with quotes (#592)

* Initial plan

* Fix ETag format to comply with HTTP standards by wrapping with quotes

Co-authored-by: overtrue <[email protected]>

* bufigx

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: overtrue <[email protected]>
Co-authored-by: overtrue <[email protected]>

* Improve lock (#596)

* improve lock

Signed-off-by: Mu junxiang <[email protected]>

* feat(tests): add wait_for_object_absence helper and improve lifecycle test reliability

Signed-off-by: Mu junxiang <[email protected]>

* chore: remove dirty docs

Signed-off-by: Mu junxiang <[email protected]>

---------

Signed-off-by: Mu junxiang <[email protected]>

* feat(append): implement object append operations with state tracking (#599)

* feat(append): implement object append operations with state tracking

Signed-off-by: junxiang Mu <[email protected]>

* chore: rebase

Signed-off-by: junxiang Mu <[email protected]>

---------

Signed-off-by: junxiang Mu <[email protected]>

* build(deps): upgrade s3s (#595)

Co-authored-by: loverustfs <[email protected]>

* fix: validate mqtt broker

* improve code for `import`

* fix

* improve

* remove logger from `rustfs-obs` crate

* remove code for config Observability

* fix

* improve code

* fix comment

* up

* up

* upgrade version

* fix

* fmt

* upgrade tokio version to 1.48.0

* upgrade `datafusion` and `reed-solomon-simd` version

* fix

* fmt

* improve code for notify webhook example

* improve code

* fix

* fix

* fmt

---------

Signed-off-by: Mu junxiang <[email protected]>
Signed-off-by: junxiang Mu <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: overtrue <[email protected]>
Co-authored-by: overtrue <[email protected]>
Co-authored-by: guojidan <[email protected]>
Co-authored-by: Nugine <[email protected]>
Co-authored-by: loverustfs <[email protected]>
2025-10-17 15:34:53 +08:00
aac9b1edb7 chore: improve event and docker-compose ,Improve the permissions of the endpoint health interface, upgrade otel from 0.30.0 to 0.31.0 (#620)
* feat: improve code for notify

* upgrade starshard version

* upgrade version

* Fix ETag format to comply with HTTP standards by wrapping with quotes (#592)

* Initial plan

* Fix ETag format to comply with HTTP standards by wrapping with quotes

Co-authored-by: overtrue <[email protected]>

* bufigx

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: overtrue <[email protected]>
Co-authored-by: overtrue <[email protected]>

* Improve lock (#596)

* improve lock

Signed-off-by: Mu junxiang <[email protected]>

* feat(tests): add wait_for_object_absence helper and improve lifecycle test reliability

Signed-off-by: Mu junxiang <[email protected]>

* chore: remove dirty docs

Signed-off-by: Mu junxiang <[email protected]>

---------

Signed-off-by: Mu junxiang <[email protected]>

* feat(append): implement object append operations with state tracking (#599)

* feat(append): implement object append operations with state tracking

Signed-off-by: junxiang Mu <[email protected]>

* chore: rebase

Signed-off-by: junxiang Mu <[email protected]>

---------

Signed-off-by: junxiang Mu <[email protected]>

* build(deps): upgrade s3s (#595)

Co-authored-by: loverustfs <[email protected]>

* fix: validate mqtt broker

* improve code for `import`

* upgrade otel relation crates version

* fix:dep("jsonwebtoken") feature = 'rust_crypto'

* fix

* fix

* fix

* upgrade version

* improve code for ecfs

* chore: improve event and docker-compose ,Improve the permissions of the `endpoint` health interface

* fix

* fix

* fix

* fix

* improve code

* fix

---------

Signed-off-by: Mu junxiang <[email protected]>
Signed-off-by: junxiang Mu <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: overtrue <[email protected]>
Co-authored-by: overtrue <[email protected]>
Co-authored-by: guojidan <[email protected]>
Co-authored-by: Nugine <[email protected]>
Co-authored-by: loverustfs <[email protected]>
2025-10-11 09:08:25 +08:00
weisdandGitHub 90f21a9102 refactor: Reimplement bucket replication system with enhanced architecture (#590)
* feat:refactor replication

* use aws sdk for replication client

* refactor/replication

* merge main

* fix lifecycle test
2025-09-26 14:27:53 +08:00
29b0935be7 RustFS rustfs-audit Complete Implementation with Enterprise Observability (#557)
* Initial plan

* Implement core audit system with multi-target fan-out and configuration management

Co-authored-by: houseme <[email protected]>

* Changes before error encountered

Co-authored-by: houseme <[email protected]>

* Complete audit system with comprehensive observability and test coverage

Co-authored-by: houseme <[email protected]>

* improve code

* fix

* improve code

* fix test

* fix test

* fix

* add `rustfs-audit` to `rustfs`

* upgrade crate version

* fmt

* fmt

* fix

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: houseme <[email protected]>
Co-authored-by: houseme <[email protected]>
2025-09-24 08:23:46 +08:00