rustfs/docs/operations/drive-timeout-tuning.md

Drive Timeout Tuning

Use this when: ListObjects/ListObjectsV2 on a large prefix fails with Io error: timeout, or RustFS runs on HDD-class, network, or throttled storage and you need to widen per-operation drive liveness budgets. Source of truth: crates/config/src/constants/drive.rs (DEFAULT_DRIVE_*_TIMEOUT_SECS, DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS), crates/config/src/constants/object.rs (DEFAULT_OBJECT_DISK_READ_TIMEOUT), crates/config/src/constants/capacity.rs (DEFAULT_CAPACITY_MAX_TIMEOUT_SECS), crates/ecstore/src/cache_value/metacache_set.rs (walk stall handling and rustfs_list_path_raw_stall_total).

This document describes the per-operation drive timeout knobs and the drive-timeout profile. It is written for operators running RustFS on slow or high-latency storage (HDD-class disks, network block devices, throttled containers) who see ListObjects/ListObjectsV2 requests fail on large prefixes, or who want to widen drive liveness budgets before a walk on a healthy-but-slow disk is treated as a stall.

If you are debugging a listing that returns fewer keys than the bucket holds, read Listing truncation and the walk stall budget first — that is the case this document exists for.

Background: what these timeouts bound

Every foreground drive operation carries a liveness budget so a hung disk fails fast instead of parking the request forever. The budget answers "is the drive still answering", not "how much total work is there" — a healthy disk that is merely busy keeps making progress and is not timed out.

The most important of these for listings is the walk stall budget. A directory walk (the filesystem enumeration behind every ListObjects) bounds each individual filesystem call — readdir, stat, xl.meta read — by the stall budget. A call that stops answering for longer than the budget fails with a drive timeout; time the walk spends blocked on a slow consumer (a client draining the listing slowly) does not count against it.

Configuration and precedence

Each knob is resolved in this order, highest priority first:

1. Its explicit per-operation environment variable (RUSTFS_DRIVE_*_TIMEOUT_SECS).
2. The legacy global fallback RUSTFS_DRIVE_MAX_TIMEOUT_DURATION (deprecated; applies to every per-operation knob that has no explicit override).
3. The drive-timeout profile default (see below).
4. The built-in default.

Values are whole seconds. Changes take effect on process restart.

Drive-timeout profile

RUSTFS_DRIVE_TIMEOUT_PROFILE selects a preset that raises several defaults at once, so slow-storage deployments do not have to set each knob individually:

Value	Effect
default	Built-in defaults (see the table below).
high_latency	Raises the default for every profile-aware knob to 60s.

Explicit per-operation overrides always win over the profile, so you can select high_latency and still pin one knob to a specific value.

Knobs

Environment variable	Default	high_latency default	Bounds
RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS	5	60	Max time a single walk filesystem call may go without answering during a listing walk. The knob for listing failures on large prefixes.
RUSTFS_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS	10	120	Max time the metacache merge consumer waits for the next visible entry from a walk reader. Values below the resolved stall timeout are clamped up to the stall timeout.
RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS	5	60	Total wall-clock timeout for a walk_dir. Retained for non-foreground callers; the foreground listing path no longer uses it (see below).
RUSTFS_DRIVE_LIST_DIR_TIMEOUT_SECS	5	60	Timeout for a standalone list_dir metadata listing.
RUSTFS_DRIVE_METADATA_TIMEOUT_SECS	5	60	Timeout for metadata reads such as read_metadata.
RUSTFS_DRIVE_DISK_INFO_TIMEOUT_SECS	5	60	Timeout for disk_info() calls.
RUSTFS_OBJECT_DISK_READ_TIMEOUT	10	60	Per-read stall budget while streaming an object body from disk.
RUSTFS_CAPACITY_STAT_TIMEOUT	3	60	Base cooperative budget for an object-capacity disk scan.
RUSTFS_CAPACITY_MAX_TIMEOUT	15	60	Upper bound for the dynamically adjusted object-capacity scan budget.
RUSTFS_DRIVE_MAX_TIMEOUT_DURATION	30	—	Deprecated global fallback for every per-operation knob without an explicit override. Prefer the per-operation knobs.

The health-transition and probe knobs (RUSTFS_DRIVE_TIMEOUT_HEALTH_ACTION, RUSTFS_DRIVE_ACTIVE_CHECK_*, RUSTFS_DRIVE_SUSPECT_FAILURE_THRESHOLD, and the returning/offline classification knobs) govern how a timeout maps to drive health state. They are out of scope here; see crates/config/src/constants/drive.rs for the full list and defaults.

Listing truncation and the walk stall budget

Symptom

ListObjects/ListObjectsV2 on a large prefix either:

• returns 500 InternalError with Io error: timeout; or
• on builds without the failure contract below, returns HTTP 200 with IsTruncated=false after fewer keys than the bucket actually holds — a silent truncation that S3 clients (mc, minio-go, SDK pagination loops) cannot detect, because IsTruncated=false is the protocol's only end-of-listing signal.

Every "missing" object remains readable by exact key via GetObject / StatObject; only the listing is affected.

Why it happens

A listing walk is bounded by the stall budget. Because a whole-directory enumeration (list_dir reading every immediate child in one pass) is bounded by that budget as a single unit, a very wide flat directory — one prefix holding hundreds of thousands or millions of immediate children — can make a single readdir exceed the budget on a perfectly healthy disk, especially on HDD-class or throttled storage. That trips a drive timeout, which the listing path escalates and surfaces to the client.

Failure contract

• A walk that stalls after streaming any entries fails as a hard drive timeout escalated on that erasure set; the client always sees an error, never a well-formed short page. Locked by list_path_raw_returns_timeout_when_producer_fails_after_partial_entry in crates/ecstore/src/cache_value/metacache_set.rs.
• The remaining 500 on a genuinely wide flat directory is an operator tunable, not a data-integrity bug: widen the stall budget as below.

Mitigation

Raise the walk stall budget, or select the high-latency profile:

# widen just the listing walk budget
-e RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS=60

# widen the metacache reader wait budget when drives keep progressing but do
# not publish visible entries quickly enough for the merge consumer
-e RUSTFS_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS=120

# or raise every profile-aware drive default at once
-e RUSTFS_DRIVE_TIMEOUT_PROFILE=high_latency

The most durable fix for pathologically wide directories is to shard keys under additional prefix levels so no single directory holds an enormous flat child set; the stall budget then never has to bound one giant readdir.

Diagnosing

Server logs at the failure show the walk timeout escalating through the listing pipeline:

WARN  Metacache reader peek timed out            state=peek_timed_out  drive=<endpoint>
ERROR Metacache listing quorum failed            state=quorum_failed

The rustfs_list_path_raw_stall_total counter (labelled by drive) increments on every walk stall, and gives you a per-drive signal that a budget is being hit before any client-visible failure. Watch it after tuning to confirm the stalls have stopped.