唐小鸭andGitHub
e11ce2f132
* fix(site-replication): route every state RMW through the locked transaction P1-15 PR2 (rustfs/backlog#1796, batch B2 of rustfs/backlog#1675), the follow-up promised by rustfs/rustfs#5882. PR1 left ~26 read-modify-write call sites on config/site-replication/state.json in the pre-transaction shape: a process-local mutex around load / mutate / save, each IO taking its own object lock. Nothing held a distributed lock across the whole sequence, so two nodes of one site still lost each other's updates, and the transitional mutex kept the old shape available to copy. Every remaining RMW now runs inside update_site_replication_state; read-only sites use load_site_replication_state, whose object read comes with the object-level read lock. SITE_REPLICATION_STATE_LOCK and its owner helper are gone, together with their architecture-guard allowlist entry and inventory row. The multi-stage flows (add / edit / peer join / peer edit / remove / rotate) keep their updated_at and pending-id CAS, but the CAS now runs inside the transaction that writes, against the state that transaction loaded. Peer probes, IAM work and fan-outs run between transactions and hold no lock at all — the add no longer blocks every writer of the site across its peer join round trips, and it re-checks the precondition right after the capability probes so the common race is rejected before any IAM write or remote join. When the add's commit CAS still fails, the error says the peers may already be joined and that re-running the add reconverges. The add adopts only the fields it computed (exhaustive destructure — adding a state field is a compile error until classified); fields owned by writers that do not bump updated_at keep their freshly loaded values. Ordering of peer-edit deliveries now rests on the generation fence landed in PR1 rather than on a guard that could never order two nodes: the add's finalize fan-out carries the generation allocated in its commit. An accepted peer join PRESERVES the applied-generation high-water marks — join fan-outs are routine (adds and rotations both deliver SRPeerJoin to existing peers), so wiping them would let stalled older edits land after any join; the unilateral-removal rejoin misfence that a wipe would have patched is pre-existing since the fence landed and needs an epoch in the fence instead. The rotation handler now takes the lifecycle guard: the background service-account reconciler runs its repair under a lifecycle try-acquire, and its pending-rotation precheck is only sound if a rotation cannot start mid-repair — an exclusion the removed process mutex used to provide as a side effect. update_site_replication_state_when_changed adds persist-or-skip so ack markers and pending-clearing paths stop rewriting the object on a miss — load-bearing, because the shared persist helper clears the whole object for a ≤1-peer pending-free state — and save_site_replication_state is now cfg(test): the pre-P1-15 shape can no longer be written in production code. No on-disk format change. Verification: cargo nextest run -p rustfs -E 'test(/admin::handlers::site_replication::/)' (181 passed); site-replication dual/three-node e2e (13 passed); cargo clippy -p rustfs --all-targets -D warnings; make pre-commit. Mutation checks: dropping the state-object lock from the boundary reds the separate-node concurrency tests; flipping a persist-or-skip miss to a persist reds test_missed_pending_clear_must_not_rewrite_the_state_object. Reviewed by three independent adversarial passes (correctness/concurrency, security/compatibility, simplicity/test-coverage); their confirmed findings are folded in. * fix(site-replication): serialize peer-join admission around its IAM write Review follow-up (overtrue): two joins accepted by the same node could interleave as "A checks a stale snapshot and pauses reading its body, B applies secret B and commits, A resumes, overwrites IAM with secret A, and A's commit is refused as superseded" — the persisted state advertised B's contract while IAM only accepted A's secret, failing every peer control-plane call. The pre-P1-15 process mutex serialized same-node joins end to end; removing it dropped that exclusion. admit_peer_join now runs the staleness check, the IAM upsert and the state commit under the lifecycle guard, with the authoritative pre-check taken against a load under that guard BEFORE IAM changes anything. The closing transaction still re-checks staleness: the guard is process-local (exactly as far as the old mutex reached) and the state-object lock arbitrates joins accepted by different nodes. The body is fully read before the guard so a stalling sender cannot block add/remove/rotate/reconciler. The IAM step is injected, and the gated-body regression test reproduces the review's ordering: join A is held mid-IAM while a newer join B arrives; B must wait at the guard, and both IAM order and the final persisted state end on B. Mutation-verified: removing the lifecycle guard from admit_peer_join turns the test red. Verification: cargo nextest run -p rustfs -E 'test(/admin::handlers::site_replication::/)' (182 passed); site-replication dual/three-node e2e (13 passed); cargo clippy -p rustfs --all-targets -D warnings; make pre-commit. * fix(site-replication): fence peer-join admission across nodes Review follow-up (overtrue, round 2): the lifecycle guard only serializes joins within one process. Node A could pass the staleness check for an older T1, node B write secret B to IAM and commit a newer T2, and node A then overwrite IAM with secret A while its own state commit is refused as superseded — state advertising T2's contract while IAM only accepts A's secret. The admission (staleness check -> IAM upsert -> state commit) now also runs under a distributed join-admission lock, a namespace-lock key with no backing object, following the repair execution lock's pattern — including its nesting of config-object locks (admission -> state), and delegating crash safety to the lock subsystem's lease expiry instead of a hand-rolled TTL. The staleness check runs against a load taken inside the lock, before IAM changes anything, so a superseded join exits without touching IAM. The closing transaction keeps its re-check for defence in depth and for old-version nodes that do not take the admission lock during a rolling upgrade (that mixed-version window keeps today's behavior and closes when the upgrade completes). admit_peer_join_across_nodes is the admission minus the process-local lifecycle guard — exactly what a second node runs — and the new separate-nodes regression test drives it directly with join A gated mid-IAM: join B must wait at the distributed lock, and both the IAM write order and the final persisted state end on B. Mutation-verified: removing the admission lock turns the test red while the same-node test (which drives the full admit_peer_join) stays green. Verification: cargo nextest run -p rustfs -E 'test(/admin::handlers::site_replication::/)' (183 passed); site-replication dual/three-node e2e (13 passed); cargo clippy -p rustfs --all-targets -D warnings; make pre-commit.
RustFS is a high-performance distributed object storage software built using Rust
Getting Started · Docs · Bug reports · Discussions
English | Simplified Chinese
RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. Along with MinIO, it shares a range of advantages such as simplicity, broad S3 API compatibility for supported features, open-source nature, support for data lakes, AI, and big data. Furthermore, it has a better and more user-friendly open-source license in comparison to other storage systems, being constructed under the Apache license. As Rust serves as its foundation, RustFS provides faster speed and safer distributed features for high-performance object storage.
Features
- High Performance: Built with Rust, ensuring speed and efficiency.
- Distributed Architecture: Scalable and fault-tolerant design for large-scale deployments.
- S3 Compatibility: Integration with common S3-compatible applications; current coverage is tracked in the S3 compatibility matrix.
- Data Lake Support: Optimized for big data and AI workloads.
- Open Source: Licensed under Apache 2.0, encouraging community contributions and transparency.
- User-Friendly: Designed with simplicity in mind, making it easy to deploy and manage.
RustFS vs MinIO
Stress test server parameters
| Type | parameter | Remark |
|---|---|---|
| CPU | 2 Core | Intel Xeon(Sapphire Rapids) Platinum 8475B , 2.7/3.2 GHz |
| Memory | 4GB | |
| Network | 15Gbp | |
| Driver | 40GB x 4 | IOPS 3800 / Driver |
https://github.com/user-attachments/assets/2e4979b5-260c-4f2c-ac12-c87fd558072a
RustFS vs Other object storage
| RustFS | Other object storage |
|---|---|
| Powerful Console | Simple and useless Console |
| Developed based on Rust language, memory is safer | Developed in Go or C, with potential issues like memory GC/leaks |
| Does not report logs to third-party countries | Reporting logs to other third countries may violate national security laws |
| Licensed under Apache, more business-friendly | AGPL V3 License and other License, polluted open source and License traps, infringement of intellectual property rights |
| S3-compatible core, with coverage tracked in the compatibility matrix | Variable S3 support and local cloud vendor coverage |
| Rust-based development, strong support for secure and innovative devices | Poor support for edge gateways and secure innovative devices |
| Stable commercial prices, free community support | High pricing, with costs up to $250,000 for 1PiB |
| No risk | Intellectual property risks and risks of prohibited uses |
Quickstart
To get started with RustFS, follow these steps:
-
One-click installation script (Option 1)
curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh -
Docker Quick Start (Option 2)
# Docker Hub (recommended)
docker run -d -p 9000:9000 -v /data:/data rustfs/rustfs:latest
# Alternative using Podman
podman run -d -p 9000:9000 -v /data:/data rustfs/rustfs:latest
- Access the Console: Open your web browser and navigate to
http://localhost:9001to access the RustFS console, default username and password isrustfsadmin. - Create a Bucket: Use the console to create a new bucket for your objects.
- Upload Objects: You can upload files directly through the console or use S3-compatible APIs to interact with your RustFS instance.
Documentation
For detailed documentation, including configuration options, API references, and advanced usage, please visit our Documentation.
Getting Help
If you have any questions or need assistance, you can:
- Check the FAQ for common issues and solutions.
- Join our GitHub Discussions to ask questions and share your experiences.
- Open an issue on our GitHub Issues page for bug reports or feature requests.
Links
- Documentation - The manual you should read
- Changelog - What we broke and fixed
- GitHub Discussions - Where the community lives
Contact
- Bugs: GitHub Issues
- Business: [email protected]
- Jobs: [email protected]
- General Discussion: GitHub Discussions
- Contributing: CONTRIBUTING.md
Contributors
RustFS is a community-driven project, and we appreciate all contributions. Check out the Contributors page to see the amazing people who have helped make RustFS better.
License
RustFS is a trademark of RustFS, Inc. All other trademarks are the property of their respective owners.