Zhengchao An and GitHub
710ae74cde
perf: add S3 operations benchmark framework ( #738 ) ( #4005 )
2026-06-28 18:02:41 +08:00
Zhengchao An and GitHub
ee82d6c026
test: add insta snapshot test for storage error display format ( #740 ) ( #4001 )
...
test: add insta snapshot test for storage error display format
Add snapshot test to detect unexpected changes in StorageError
display format. This catches output format regressions that
traditional assert tests might miss.
Refs #740
2026-06-28 16:10:07 +08:00
27468ebfa9
feat(get): consolidate GET performance optimization ( #3972 )
...
* feat(get): consolidate GET performance optimization
Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.
## Changes
### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support
### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
- delete_marker_votes tracking
- requested_version_id and matching_version_votes tracking
- version_early_stop_decision() method
- 6 new tests for version early-stop scenarios
### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default
### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration
### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)
### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label
### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants
### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs
### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |
## Rollback
All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false
Co-Authored-By: heihutu <[email protected] [email protected] [email protected]
2026-06-28 07:14:07 +08:00
512418cda9
fix(ecstore): replace unbounded metadata cache with moka ( #743 ) ( #3970 )
...
fix(ecstore): replace unbounded metadata cache with moka
Replace the manual Arc<RwLock<HashMap>> metadata cache with
moka::future::Cache, which provides:
- Built-in LRU eviction when max_capacity is reached
- Automatic TTL expiry via time_to_live (250ms)
- Lock-free concurrent reads
- Non-blocking invalidation
Fixes the memory leak risk from unbounded HashMap and the
all-or-nothing eviction logic that cleared all entries at once.
Closes #743
Co-authored-by: houseme <[email protected]
2026-06-28 03:01:32 +08:00
houseme and GitHub
58b76a3d45
feat(get): add codec engine ab matrix ( #3940 )
...
* upgrade version
* feat(get): add codec engine ab matrix
* chore(get): drop unrelated dependency drift
* upgrade version
* upgrade version
* fix cargo deny
2026-06-27 13:27:16 +08:00
Zhengchao An and GitHub
840d21d201
fix(deps): remove vulnerable thrift dependency ( #3926 )
2026-06-27 00:16:51 +08:00
cxymds and GitHub
30957e2b51
fix(ecstore): throttle data movement under read pressure ( #3906 )
2026-06-26 17:48:09 +08:00
cxymds and GitHub
90638bfc19
fix(heal): add backpressure to repair admission ( #3900 )
2026-06-26 15:19:37 +08:00
houseme and GitHub
f7a9a7142b
chore: update deps and e2e cache action ( #3885 )
...
* chore(deps): update workspace crate versions
* ci: update e2e s3tests cache action
2026-06-26 04:19:28 +08:00
houseme and GitHub
623fc801f1
feat(replication): support custom TLS for bucket targets ( #3825 )
...
* feat(replication): support insecure https bucket targets
* feat(replication): support custom ca bucket targets
* test(replication): satisfy e2e clippy for TLS helpers
* fix(replication): avoid native root panic for custom trust stores
* test(replication): decouple private IP target test from TLS roots
* test(replication): use target TLS client in private IP unit test
2026-06-24 20:23:20 +08:00
houseme and GitHub
0a00d8d500
perf(server): lighten internode data-plane stack ( #3735 )
...
* refactor(server): split internode dispatch scaffold
* test(server): cover internode dispatch prefix split
* refactor(server): name internode stack boundaries
* perf(server): skip internode request logging layer
* perf(server): skip internode trace layer
* perf(server): use lite internode request context
* feat(metrics): track internode rpc duration
* feat(ecstore): add put object stage summary logs
* test(metrics): update internode descriptor expectations
* fix(server): tighten internode path matching
* fix(pr): address review follow-up comments
* style(ecstore): simplify commit tail duration field
* refactor(ecstore): group put stage summary fields
* refactor(ecstore): inline put stage summary log
* fix(s3): return storage class for object attributes
* merge: sync latest main and resolve object attributes conflict
* fmt
* fix(server): remove duplicate rpc imports
* build(deps): bump memmap2 for RUSTSEC-2026-0186
* fix(s3select): align object_store with datafusion
* chore(deps): prune workspace dependencies
* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization
Separate fuzz harness compilation from execution to eliminate redundant
builds across targets. Introduce matrix-based parallel execution for
PR smoke and nightly fuzz jobs.
Changes:
- Split CI workflow into `fuzz-build` (compile once) and matrix run jobs
(`pr-fuzz-smoke`, `nightly-fuzz-corpus`) that execute targets in parallel
- Add `BUILD_ONLY` mode to run_ci_targets.sh / run_nightly_targets.sh
- Add run_single_target.sh for matrix jobs (no build phase)
- Optimize `local_metadata` fuzz target: reduce prefix iterations from
8-10 (4 functions each) to 5 critical prefixes (parser-only), cutting
per-iteration cost by ~3-5x
- Move archive path validation (`validate_extract_relative_path`,
`normalize_extract_entry_key`) from `rustfs` to `rustfs-utils::path`,
eliminating `rustfs` binary crate dependency from fuzz harness
- Remove `rustfs` from fuzz/Cargo.toml (drops significant transitive deps)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with new workflow and script documentation
Expected CI improvement: PR smoke wall-clock from ~120min (frequent
timeout) to ~40min; nightly from ~180min to ~60min.
* refactor(fuzz): consolidate scripts and fix prefix test alignment
Replace three duplicated shell scripts (run_ci_targets.sh,
run_nightly_targets.sh, run_single_target.sh) with a single
parameterized run.sh that supports BUILD_ONLY, SKIP_BUILD, and
MAX_TOTAL_TIME environment variables.
Fix local_metadata fuzz target prefix testing: replace always-true
'len > 0' guard with lengths aligned to xl.meta binary layout
(4/5/8/12 bytes for magic+version+header fields). Remove redundant
empty-slice test.
Hoist RUSTFLAGS to workflow top-level env to eliminate per-job
duplication. Update README with unified script documentation.
Net: -118 lines, zero functionality loss.
* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization
Restructure fuzz CI workflow to eliminate redundant compilation and
run targets in parallel via matrix strategy.
Workflow changes:
- Split into fuzz-build (compile once) and matrix run jobs
- PR smoke: 3 targets parallel, 60s each, timeout 30min (was 120min)
- Nightly: 3 targets parallel, 300s each, timeout 60min (was 180min)
- Pass compiled harness via actions/artifact between jobs
- Hoist RUSTFLAGS to workflow top-level env
Script consolidation:
- Replace 3 duplicated scripts with single parameterized run.sh
- Supports BUILD_ONLY, SKIP_BUILD, MAX_TOTAL_TIME env vars
Target optimizations:
- Remove rustfs binary crate from fuzz dependencies (was pulling
979 transitive deps); move archive path validation to rustfs-utils
- Optimize local_metadata: reduce prefix iterations from 8-10x4
calls to 5 prefixes with parser-only (no decompress), aligned
with xl.meta binary layout (4/5/8/12 bytes)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with unified script documentation
Expected: PR smoke wall-clock from ~120min (frequent timeout)
to ~40min; nightly from ~180min to ~60min.
* fix(rpc): resolve internode metrics via app context
* fmt
* ci: speed up fuzz smoke artifact restore
---------
Signed-off-by: houseme <[email protected]
2026-06-23 21:36:39 +08:00
houseme and GitHub
46775ae019
feat: harden runtime capability snapshots ( #3784 )
...
* feat(admin): expose runtime capability snapshots
* feat(runtime): refine workload admission snapshots
* test(ci): align architecture migration checks
* build(deps): bump memmap2 for RUSTSEC-2026-0186
* build(deps): refresh cargo deny lockfile
2026-06-23 18:10:49 +08:00
houseme and GitHub
2f85ef4455
fix(security): extend outbound egress guard ( #3744 )
2026-06-22 19:20:31 +08:00
dependabot[bot] and GitHub
5a601869f6
chore(deps): bump sysinfo from 0.39.3 to 0.39.4 in the dependencies group ( #3712 )
2026-06-22 09:53:57 +08:00
houseme and GitHub
9e6d0c7292
chore(deps): refresh workflow actions and crate pins ( #3696 )
2026-06-21 20:29:17 +08:00
安正超 and GitHub
08b8f58e64
refactor: prune notify ecstore object alias ( #3620 )
2026-06-19 20:11:56 +08:00
houseme and GitHub
8cf3c0bfbd
fix(ecstore): support MinIO DARE fixture compatibility ( #3590 )
2026-06-19 10:13:39 +08:00
houseme and GitHub
53ff19dd2d
fix(kms): harden local master key storage ( #3589 )
2026-06-19 08:50:35 +08:00
安正超 and GitHub
b1c6578df1
refactor: narrow test harness compatibility surfaces ( #3592 )
2026-06-19 07:10:52 +08:00
安正超 and GitHub
c28fee0013
refactor: continue storage api contract cleanup ( #3580 )
...
* refactor: move delete object contracts to storage api
* refactor: narrow store api compatibility exports
* refactor: route table catalog test through storage compat
2026-06-18 22:42:02 +08:00
houseme and GitHub
8cf11af649
fix(security): add shared outbound egress guard ( #3567 )
...
* fix(security): block unsafe outbound webhook targets
* chore: keep issue-3557 plan local only
2026-06-18 15:30:09 +08:00
安正超 and GitHub
54bbd05b25
refactor: move object list helper contracts ( #3546 )
2026-06-18 06:09:33 +08:00
houseme and GitHub
87a3d107d6
test(property): add path validator invariants ( #3541 )
...
test(property): add path validator invariants for #3525
2026-06-18 02:14:24 +08:00
houseme and GitHub
bc769948ab
test(property): add erasure recoverability invariants ( #3540 )
2026-06-18 01:34:07 +08:00
安正超 and GitHub
53337a6f71
refactor: move HTTP range helper contracts ( #3533 )
2026-06-18 00:47:23 +08:00
houseme and GitHub
cc84d9065e
test(property): add filemeta invariants ( #3536 )
...
* test(property): add filemeta invariants for #3523
* docs: relocate issue-3518 property testing plan
* docs: keep issue-3518 property plan local only
2026-06-18 00:36:55 +08:00
houseme and GitHub
841f86fe90
test(property): expand parser invariants ( #3534 )
...
* test(property): harden parser invariants for #3522
* test(property): satisfy clippy for #3522
2026-06-17 23:52:30 +08:00
安正超 and GitHub
919deeb816
refactor: move object option helper contracts ( #3521 )
2026-06-17 22:56:10 +08:00
65c724c786
feat(table-catalog): add manifest reachability cleanup ( #3484 )
...
Co-authored-by: Henry Guo <[email protected]
2026-06-15 22:29:23 +08:00
houseme and GitHub
adec195486
chore(scanner): enable pulsar telemetry feature ( #3481 )
2026-06-15 18:43:14 +08:00
fbf1f9e5b8
build(deps): bump brotli from 8.0.3 to 8.0.4 in the dependencies group ( #3458 )
...
* build(deps): bump brotli from 8.0.3 to 8.0.4 in the dependencies group
Bumps the dependencies group with 1 update: [brotli](https://github.com/dropbox/rust-brotli ).
Updates `brotli` from 8.0.3 to 8.0.4
- [Release notes](https://github.com/dropbox/rust-brotli/releases )
- [Commits](https://github.com/dropbox/rust-brotli/compare/8.0.3...8.0.4 )
---
updated-dependencies:
- dependency-name: brotli
dependency-version: 8.0.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
...
Signed-off-by: dependabot[bot] <[email protected] [email protected] [email protected]
2026-06-15 12:06:01 +08:00
036741cb1c
refactor(request-id): align contracts and lock field names ( #3454 )
...
* refactor(request-id): align header log and trace contracts
* test(contract): lock external request-id field names
* chore(deps): drop unused rustfs-ecstore links
Co-Authored-By: heihutu <[email protected]
2026-06-15 01:59:11 +08:00
046d5386ba
feat: stream object zip downloads ( #3380 )
...
* feat: stream object zip downloads
* fix: stream zip downloads page by page
* fix: prepare zip downloads before streaming
---------
Signed-off-by: houseme <[email protected] [email protected]
2026-06-14 14:26:44 +08:00
houseme and GitHub
e8012bd1ba
refactor(logging): normalize admin telemetry and error messages ( #3430 )
2026-06-14 13:27:10 +08:00
houseme and GitHub
0d68851f7a
refactor(logging): standardize rustfs runtime events ( #3396 )
...
* fix(rpc): adjust log levels and reduce noise in storage RPC layer
Issue #682 : pool rebalance normal flow logs were using warn! instead
of info!. Additionally, several high-frequency RPC entry points (ping,
write_stream, read_at, walk_dir) were logging at info! level, causing
unnecessary log noise in production.
Changes:
- load_rebalance_meta: warn! → info! for normal flow (3 occurrences)
- ping body decode: info! → debug! (fires on every health check)
- write_stream/read_at/walk_dir entry: info! → debug!
- metrics.rs: lowercase error messages, add structured error field
- http_service.rs: add structured error field to walk_dir failure
- Add tracing::instrument fields for start_rebalance context
* refactor(logging): standardize rustfs runtime events
* build(deps): bump workspace dependency versions
* build(deps): pin time to 0.3.47
* build(deps): update postgres client versions
2026-06-13 08:39:47 +08:00
安正超 and GitHub
434b71e569
feat(extension): add admin views ( #3386 )
2026-06-12 15:54:42 +08:00
安正超 and GitHub
1e7c376586
feat(targets): add extension schema adapter ( #3385 )
2026-06-12 15:27:55 +08:00
安正超 and GitHub
7d6d56a547
feat(extension): add schema contracts ( #3384 )
2026-06-12 15:07:17 +08:00
安正超 and GitHub
7146c893cb
refactor(storage-api): remove legacy bucket DTO re-export ( #3364 )
2026-06-11 21:24:56 +08:00
安正超 and GitHub
2205991180
refactor(config): extract server config model ( #3351 )
2026-06-11 16:11:17 +08:00
houseme and GitHub
7d38b0cf90
refactor(obs): unify local and otlp log output ( #3352 )
...
wip(obs): start lg-002 output unification
2026-06-11 16:10:54 +08:00
houseme and GitHub
8d337c4e87
feat(obs): add logging redaction guard rails ( #3349 )
...
* feat(obs): add logging redaction guard rails
* chore(docs): keep logging plans out of lg-001 pr
* test(obs): guard against unmasked access key logs
* test(obs): enforce masked access key log patterns
2026-06-11 15:24:32 +08:00
houseme and GitHub
ee96e194a3
build(deps): bump s3s for header parsing fix ( #3344 )
2026-06-11 04:28:36 +00:00
安正超 and GitHub
87968275a7
refactor(storage): route maintenance inventory reads ( #3337 )
2026-06-11 10:32:48 +08:00
安正超 and GitHub
94c53af264
refactor(storage): use admin API for observability reads ( #3335 )
2026-06-11 08:13:15 +08:00
安正超 and GitHub
8a90bda16a
refactor(admin): use storage admin drive counts ( #3332 )
2026-06-11 06:36:18 +08:00
安正超 and GitHub
d9ddd1bedc
feat(storage-api): add disk inventory contract ( #3330 )
2026-06-10 16:33:31 +00:00
bb5d9565a6
feat(storage-api): add bucket DTO contract ( #3314 )
...
* feat(storage-api): add bucket DTO contract
* ci(build): increase workflow timeout to 90 minutes
---------
Co-authored-by: copilot-swe-agent[bot] <[email protected] [email protected]
2026-06-10 07:16:14 +00:00
b9c924a6ed
build(ci): update macOS runner and bump regex/uuid ( #3318 )
...
* ci(build): use macos-26-intel for x86_64 job
* build(deps): bump regex and uuid
* ci(build): disable cross for macos x86_64 job
---------
Co-authored-by: majinghe <[email protected]
2026-06-10 04:42:10 +00:00
houseme and GitHub
68400933b5
chore(release): prepare 1.0.0-beta.8 ( #3317 )
...
* chore(release): prepare 1.0.0-beta.8
* chore(release): align release assets for 1.0.0-beta.8
2026-06-10 04:34:39 +00:00
