rustfs/crates/e2e_test

e2e_test

End-to-end test suite for RustFS. Each test spawns a real rustfs binary (built on demand from the workspace) and drives it over the network with the AWS SDK (aws-sdk-s3), raw HTTP (reqwest / awscurl), or a protocol client (FTPS / WebDAV / SFTP). This is the black-box integration layer: exhaustive end-to-end behavior lives here, unit behavior stays in the source crates (see AGENTS.md).

The harness lives in src/common.rs (single-node + cluster environments, S3 client construction, awscurl helpers) and src/chaos.rs (in-process disk fault injection). Crate-wide test conventions and environment-safety rules are in AGENTS.md; this file is the contributor guide.

Module map (~50 modules)

Registered in src/lib.rs. Grouped by concern:

Group	Location	What it covers
functional	top-level *_test.rs	S3 data plane: list_objects_*, copy_object_*, delete_objects_versioning, head_object_*, checksum_upload, compression, content_encoding, special_chars, leading_slash_key, create_bucket_region, quota, data_usage, snowball_auto_extract, mc_mirror_small_bucket, archive_download_integrity, version_id_regression, delete_marker_migration_semantics
object_lock	src/object_lock/	Retention / legal-hold / WORM semantics
kms	src/kms/	SSE-S3 / SSE-KMS / SSE-C, local + Vault backends, multipart encryption. Own guide: src/kms/README.md
policy	src/policy/, existing_object_tag_policy_test, bucket_policy_check_test, anonymous_access_test, security_boundary_test, multipart_auth_test	IAM / bucket-policy / STS session policy, policy variables, anonymous access, DoS/SSRF boundaries. Own guide: src/policy/README.md
protocols	src/protocols/	FTPS, WebDAV, SFTP compliance. Fixed ports, own guide: src/protocols/README.md
reliant	src/reliant/	Tests that reuse an externally started server (SQL/select, conditional writes, lifecycle, deleted-object reads, node-interact). Run via scripts/run_e2e_tests.sh; see src/reliant/README.md
cluster	cluster_concurrency_test, stale_multipart_cleanup_cluster_test, namespace_lock_quorum_test, admin_timeout_regression_test, object_lambda_test, replication_extension_test	Multi-node scenarios via RustFSTestClusterEnvironment
chaos / reliability	src/chaos.rs, reliability_disk_fault_test, heal_erasure_disk_rebuild_test, server_startup_failfast_test	Disk offline/replace/corrupt, EC rebuild, heal, fail-fast startup

How to run

All commands assume repo root. cargo test triggers an on-demand build of the rustfs binary from src/common.rs (rustfs_binary_path) on first use — the first invocation is slow, later ones reuse the binary.

# Whole crate (default = ignored tests skipped)
cargo nextest run -p e2e_test

# One module
cargo nextest run -p e2e_test -E 'test(list_objects_v2_pagination_test)'

# PR smoke subset (see "CI smoke subset" below)
cargo nextest run --profile e2e-smoke -p e2e_test

# ILM serial lane — ignored lifecycle tests, single-threaded (mirrors CI)
cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \
  -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'

# Protocols suite — fixed ports, MUST be single-threaded, gated by build features
RUSTFS_BUILD_FEATURES=ftps,webdav,sftp \
  cargo test -p e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture

The protocols suite has its own contract (fixed bind ports 9022–9301, --test-threads=1, feature-gated scheduling) documented in src/protocols/README.md. RUSTFS_BUILD_FEATURES selects which features the spawned binary is built with; leave it unset to run every protocol entry.

#[ignore] semantics

Ignored tests are excluded from the default cargo nextest run pass because they need something the default runner does not provide. Do not maintain a static count here — it rots (the set shrinks as ci-13 / ilm-3 activate suites). Read the live sources instead:

rg -n '#\[ignore' crates/e2e_test/src   # every ignore + its reason string

The reason string on each attribute is the classifier. Current classes:

• Needs a pre-started server — "requires running RustFS server at localhost:9000" / "Connects to existing rustfs server". These are the reliant/* and policy/test_runner tests; start a server first (e.g. scripts/run_e2e_tests.sh) or use --run-ignored.
• Heavy / external tool — "Starts a rustfs server; enable when running full E2E", "requires awscurl and spawns a real RustFS server". Spawn their own server and/or need awscurl on PATH.
• Serial / global-state (ILM lane) — lifecycle tests bind fixed ports and share process-global singletons; run via the ILM serial lane above.

How to add a test

Single-node (the common case)

Use RustFSTestEnvironment from src/common.rs. It picks a random free port and a unique temp dir per instance, so tests are parallel-safe by construction and clean up on Drop:

use crate::common::{RustFSTestEnvironment, TEST_BUCKET};

#[tokio::test]
async fn my_case() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let mut env = RustFSTestEnvironment::new().await?;
    env.start_rustfs_server(vec![]).await?;       // waits for readiness
    let client = env.create_s3_client();          // aws-sdk-s3 Client
    env.create_test_bucket(TEST_BUCKET).await?;
    // ... drive `client` ...
    Ok(())
}

Register the module in src/lib.rs under #[cfg(test)].

Cluster

Use RustFSTestClusterEnvironment::new(node_count) then .start(); it spawns node_count servers over a shared erasure set and hands out per-node S3 clients via create_s3_client(idx) / create_all_clients(). See cluster_concurrency_test.rs and namespace_lock_quorum_test.rs for patterns.

Fixture / helper inventory (src/common.rs)

Helper	Purpose
RustFSTestEnvironment::new / with_address	Single-node env; random or fixed address
start_rustfs_server / _with_env / _without_cleanup	Spawn the server (optional extra args / env vars / no pre-cleanup)
wait_for_server_ready	Poll readiness before issuing requests
create_s3_client / create_test_bucket / delete_test_bucket	aws-sdk-s3 client + bucket lifecycle
find_available_port	Random free port (isolation primitive)
rustfs_binary_path / _with_features	Locate/build the binary; honors RUSTFS_BUILD_FEATURES
requested_rustfs_build_features / rustfs_build_feature_enabled	Feature-gate a test to what the binary was built with
awscurl_available + execute_awscurl / awscurl_post / _get / _put / _delete / awscurl_post_sts_form_urlencoded	Admin/STS API calls via awscurl (skip gracefully when absent)
replication_fast_env	Env vars that shrink replication timers (from repl-4); pass to start_rustfs_server_with_env
local_http_client / init_logging	Loopback HTTP client; idempotent tracing init
RustFSTestClusterEnvironment (new/start/start_node/stop_node/create_all_clients)	Multi-node harness
Constants: DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, TEST_BUCKET, ENV_RUSTFS_BUILD_FEATURES	Shared credentials / bucket name / env-var name

Fault injectors live in src/chaos.rs: DiskFaultHarness (take_disk_offline, bring_disk_online, replace_disk_with_empty, corrupt_object_shard, object_metadata_exists_on_disk, kill_server / restart_server) plus signed_admin_post.

Isolation rules

• Port: never hard-code a port for single-node tests — new() allocates a random one. Fixed ports (protocols, ILM lane) force --test-threads=1 / a serial CI lane.
• Temp dir: each env owns a temp dir cleaned on Drop; do not write under a shared path.
• Orphans: RustFSTestEnvironment kills its child on Drop, but a panicked or kill -9'd run can leak a rustfs process holding a port — see Troubleshooting.

#[serial] vs nextest reality

serial_test's #[serial] uses an in-process mutex. Under nextest each test runs in its own process, so #[serial] does not serialize across tests there — see the header of .config/nextest.toml. Real cross-test serialization comes from a nextest test-group (max-threads = 1) or a -j1 CI lane. Single-node e2e tests should instead be parallel-safe by construction (random port + isolated temp dir) and need no serialization.

CI map

e2e_test is excluded from the main cargo nextest run --profile ci --all pass (.github/workflows/ci.yml line 158, --exclude e2e_test) — the whole crate is too slow to gate every PR. Subsets join CI through the nextest profile system only (never as ad-hoc jobs):

Suite	Runs where	Status
Smoke subset (e2e-smoke profile)	e2e-tests job, every PR	Active (backlog#1149 ci-4)
s3s-e2e black-box	e2e-tests + e2e-tests-rio-v2 jobs	Active (external conformance tool)
ILM / lifecycle (ignored)	test-ilm-integration-serial lane, -j1	Active (backlog#1148 ilm-1)
KMS suite	—	Not in CI yet (backlog#1149 ci-5)
Protocols (FTPS/WebDAV/SFTP)	—	Not in CI yet (backlog#1149 ci-7)
Replication (fast subset)	e2e-smoke profile, e2e-tests job, every PR	Active (backlog#1147 repl-1)
Replication (slow + dual-node)	e2e-repl-nightly profile, scheduled workflow	Active (backlog#1147 repl-1)
reliant/* (pre-started server)	—	Manual only

Links: ci.yml e2e-tests (line 347), test-ilm-integration-serial (line 196). The e2e-smoke default-filter in .config/nextest.toml is the single wiring mechanism — extend that filter (or add a sibling profile) to admit more tests; do not add e2e jobs to ci.yml. repl-1 / ilm-3 are landing in parallel and may add lanes; keep the table above easy to extend.

Troubleshooting

Reproduce a CI failure locally — run the exact profile/lane:

# Smoke (e2e-tests job) — includes the 20 fast replication tests
cargo nextest run --profile e2e-smoke -p e2e_test
# Replication nightly lane (16 slow + dual-node tests; install awscurl for the
# STS dual-node test, else it skips gracefully)
cargo nextest run --profile e2e-repl-nightly -p e2e_test
# ILM serial lane
cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \
  -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'
# s3s-e2e black box
./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs-e2e-data

Stale binary. Tests build the rustfs binary once and reuse it. To avoid rebuilding while iterating on tests, common.rs reuses an existing binary when running inside the e2e test process even if sources changed (can_reuse_inside_e2e, src/common.rs line 98). Downside: if you changed server code, force a rebuild with cargo build -p rustfs (or touch a source file outside the reuse window) before re-running, or CI's freshly built artifact will diverge from your local one.

Port already in use / orphan processes. A hard-killed run can leak a rustfs child holding its port. Find and kill it:

pkill -f 'target/debug/rustfs' ; pkill -f 'target/release/rustfs'

The s3s-e2e CI job selects a random RUSTFS_TEST_PORT (see the e2e-tests job) to dodge this; local single-node tests already use random ports, so a lingering orphan is usually the cause of a spurious bind failure.

awscurl not found. awscurl-dependent tests skip gracefully with a visible log line (awscurl_available()); install awscurl to actually run them.

Related

• Crate rules & environment safety: AGENTS.md
• Sub-suite guides: src/kms/README.md, src/policy/README.md, src/protocols/README.md, src/reliant/README.md
• Authoritative per-module counts: docs/testing/e2e-suite-inventory.md
• Test pyramid & flake policy: docs/testing/README.md

CI smoke subset (--profile e2e-smoke)

A subset of this crate runs on every PR via the e2e-tests job:

cargo nextest run --profile e2e-smoke -p e2e_test

The selection lives in .config/nextest.toml under [profile.e2e-smoke] (default-filter). That filter is the single wiring mechanism for e2e tests in CI — extend it (or add a sibling profile) instead of adding new e2e jobs to ci.yml.

Admission criteria for the smoke subset

A test module may join the smoke filter only if every test in it is:

1. Fast — single-digit seconds per test; the whole subset must keep the e2e-tests job ≤ 20 minutes.
2. Single-node — spawns its own server via RustFSTestEnvironment/start_rustfs_server on a random port with an isolated temp dir. No RustFSTestClusterEnvironment, no fixed ports.
3. Dependency-free — no pre-started server at localhost:9000, no Vault, no fixed protocol ports. Tools that may be absent on the runner (e.g. awscurl) are acceptable only when the test skips gracefully with a visible log line (see bucket_policy_check_test.rs).
4. Not #[ignore] — ignored tests are activation work (backlog#1149 ci-13 / backlog#1148 ilm-3), not smoke candidates.

Note on #[serial]: nextest runs each test in its own process, so serial_test's in-process mutex does not serialize across tests there (see the header of .config/nextest.toml). Smoke tests must therefore be parallel-safe by construction (random port + isolated temp dir), which the current subset is.

Authoritative test inventory

docs/testing/e2e-suite-inventory.md records the per-module test counts as listed by cargo nextest list -p e2e_test. Regenerate it when adding or moving e2e tests so acceptance numbers in the test-strategy issues (backlog#1147–#1155) stay auditable.