Commit Graph
143 Commits
Author SHA1 Message Date
Zhengchao AnandGitHub d22cb5d07a fix: resolve release-blocking integration failures (#6320)
* fix: resolve release-blocking integration failures

* fix: satisfy stable clippy lints

* fix: satisfy Rust 1.98 CI lints
2026-08-21 06:39:29 +08:00
762919b1ba perf(scanner): reduce per-object allocation churn (#6318)
Co-authored-by: heihutu <[email protected]>
2026-08-20 16:47:48 +00:00
114bf5148c refactor(heal): prune statistics label helpers (#6312)
Co-authored-by: heihutu <[email protected]>
2026-08-20 22:43:59 +08:00
housemeandGitHub 8bd2d5d967 perf(heal): reduce scheduler skip heap churn (#6311) 2026-08-20 21:47:11 +08:00
efbef700ea fix(heal): emit MRF repair notices on completion (#6309)
Move MRF repaired-event fan-out from admission to successful terminal completion so scanner pending-heal ledgers only clear after the canonical heal task actually finishes. Track notice ownership across duplicate admission, retry merge, cancellation, and queue displacement.

Co-authored-by: heihutu <[email protected]>
2026-08-20 21:23:29 +08:00
a247c79359 perf(heal): trim scanner and heal queue hot paths (#6307)
Cache heal queue dedup keys, avoid retry request double construction, clear task aliases after terminal completion, and age out stale scanner pending-heal ledger entries during retry sweeps.

Co-authored-by: heihutu <[email protected]>
2026-08-20 20:28:22 +08:00
2b9dcc646f refactor(heal): split manager.rs queue/scheduler/scan children (#6303)
Split the 6723-line manager.rs (44% inline tests) into a canonical
manager.rs + manager/ module tree with zero behavior change:

- manager.rs (~1830): HealManager and HealState, HealConfig, task
  report/snapshot types, overlap policy, admission classification and
  queue admission, submit paths, task-state queries, and the
  statistics surface
- manager/queue.rs (~450): the priority heal queue, its per-key dedup
  index, and the queue bookkeeping structs
- manager/scheduler.rs (~620): start_scheduler and the
  process_heal_queue consumption loop with its skip/metric helpers
- manager/auto_scan.rs (~550): the automatic disk scanner
- manager/unclean_shutdown.rs (~390): unclean-shutdown recovery and
  its durable replacement-intent helpers
- manager/tests.rs (~2970): the inline test module as a child module

All module paths are unchanged. The queue structs' fields and the
cross-module helpers gain pub(super), whose scope equals the old
single-module privacy domain; HealManager's private fields stay in the
root and remain reachable from child impl blocks. Code is moved
verbatim apart from those markers, heal-level super:: path fixes for
the unclean-shutdown move, per-module import headers, and rustfmt
re-wraps.

The logging-guardrail rule for the manager demote_to_debug_when! count
now sums manager.rs with its manager/*.rs children, since one
scheduler site moved with process_heal_queue; the >= 6 threshold is
unchanged and the forbidden admission info!/warn! pattern check keeps
targeting the root admission code.

Co-authored-by: heihutu <[email protected]>
2026-08-20 19:12:07 +08:00
housemeandGitHub 0d129ec4e7 refactor(heal): split task.rs per heal kind (#6293) 2026-08-20 12:24:16 +08:00
e6fc661162 refactor(heal): split resume.rs into focused child modules (#6290)
Split the 4242-line resume.rs (46% inline tests) into a canonical
foo.rs + foo/ module tree with zero behavior change:

- resume.rs (~1020): state file constants, PersistThrottle, ResumeState,
  ResumeManager core (constructors, load/discovery, progress mutators,
  ordinary persistence) plus root re-exports
- resume/replacement.rs (~690): replacement-intent/proof types and the
  ResumeManager replacement-lifecycle methods
- resume/checkpoint.rs (~350): ResumeCheckpoint + CheckpointManager
- resume/utils.rs (~310): ResumeUtils statics
- resume/tests.rs (~1980): the inline test module as a child module

All module paths are unchanged (heal::resume::CheckpointManager and
friends resolve through root re-exports), so no consumer inside or
outside the crate changes. Items defined in child modules keep
module-private visibility; only the ten cross-module helpers gain
pub(super), which is not part of the crate API. Code is moved verbatim
apart from those visibility markers, four super::storage_api path
fixes, and the new per-module import headers.

Co-authored-by: heihutu <[email protected]>
2026-08-20 12:11:35 +08:00
898aa4db95 chore: adjudicate the last 18 bare dead_code allows in the library crates (#6265)
* chore: adjudicate the last 18 bare dead_code allows in the library crates

Finishes backlog#1823 step 10 outside `rustfs/src` and `protocols`: config, s3select-query, common, madmin, heal, ecstore, signer and notify. Stripped first, then clippy asked which the compiler actually missed — 8 of the 18 were inert.

Seven items are deleted, each checked by grep as well as by clippy:

- `common/last_minute.rs`'s private `TimedAction` (with its impl) and `SizeCategory` (with its `Display` impl). The file's public surface — `AccElem`, `LastMinuteLatency` — stays; ecstore consumes it.
- `s3select-query`'s three `with_*` builders. `DefaultLogicalOptimizer::with_optimizer_rules` looks used, but the call in the same file is `SessionStateBuilder::with_optimizer_rules` from DataFusion; the local methods have no callers.
- `heal/manager.rs`'s `contains_key`. Its six apparent references are all `HashMap::contains_key`.

Three keep their code:

- `heal/storage.rs`'s `Test` variant is constructed by the `#[cfg(test)] test()` helper, which the lib target cannot see, so it takes a reasoned allow.
- `signer`'s `STREAMING_PAYLOAD_HDR` and `try_build_chunk_string_to_sign` gain the `_` prefix instead. That file already marks deliberately-unheld code that way — `_STREAMING_TRAILER_HDR`, `_PAYLOAD_CHUNK_SIZE`, and `_try_build_chunk_signature`, which is the only caller of that function. Following the existing convention removes the allow without an attribute.

`protocols` keeps its four; that crate needs `--features swift,sftp` to compile fully and is verified differently. The four `#![allow(dead_code)]` in `e2e_test` are module-root blankets in test-support files, which belong to steps 1-5 rather than step 10.

Refs backlog#1823

* chore(e2e_test): adjudicate the two dead_code allows the lib test target still needs

`cargo clippy --all-targets` compiles e2e_test's lib test target, which the earlier pass did not cover, so these two removals only surfaced in CI.

test_large_multipart_upload's allow was load-bearing: its call site in test_local_kms_multipart_upload is commented out behind "TODO: Re-enable after fixing streaming encryption issues with large files". The allow comes back with the reason string this batch uses everywhere else, so the next reader sees why it is parked instead of deleting a test we intend to run again.

TestDefinition.category was the opposite: written at all six definitions, read nowhere, and its enum's impl block is empty. The live copy of that type is crates/e2e_test/src/kms/test_runner.rs, which has an as_str; the policy copy is a vestige of it. Dropping the field, the enum, and the constructor parameter leaves the runner unchanged — it dispatches on name and filters on is_critical.

Verification: cargo clippy --all-targets -- -D warnings (workspace, the CI command) and cargo fmt --all --check both pass.

---------

Co-authored-by: houseme <[email protected]>
2026-08-19 17:55:58 +00:00
b1b4e443b2 perf(heal,scanner): single-flight MRF producers per detection event (#6282)
Two producer paths double-booked the same damage across repair records
(backlog#1894 axis A):

- The scanner's corrupt-metadata branch fired a durable MRF journal
  intent, an immediate High heal request, and a pending-ledger entry for
  the same object. When the MRF intent is accepted into the channel it
  already covers the repair durably (the consumer files a High Metadata
  heal and the journal replays it across restarts), so the immediate
  request and ledger entry are dropped in that case; on delivery failure
  (feature disabled, channel uninitialized, or full) the old immediate
  request + ledger path runs unchanged, keeping the repair safety net.
- The read path filed a journal intent before the read-repair
  reservation check, so a burst of reads failing on one object booked a
  journal record per retry. The intent now rides the submission: it is
  filed only when the sighting wins the dedup TTL, next to the Low
  request, via a new optional mrf_intent field on
  ReadRepairHealSubmission (None keeps the historical no-intent
  behavior for the other read-repair call sites).

Manager dedup-key semantics are untouched; the fix is that competing
producers stop double-booking. With RUSTFS_HEAL_MRF_ENABLE off both
paths behave exactly as before.

Co-authored-by: heihutu <[email protected]>
2026-08-19 17:32:16 +00:00
d6efb65588 feat(heal,scanner): best-effort repaired notices from the MRF consumer (#6283)
The scanner's pending-heal ledger and the MRF journal tracked the same
damaged objects with no cross-talk: once the consumer landed an intent
with the heal manager, the ledger's retry entry for that target kept
re-submitting a heal the manager already owned (backlog#1894 axis B).

Fan the acceptance out: both dispatch sites in the MRF queue (the live
consumer and the startup replay) record a compact MrfRepairedEvent
(bucket, object, version bytes) in a bounded process-wide ring owned by
rustfs-common. The scanner drains its own bucket's notices at the top
of retry_pending_scanner_heals and clears the matching Object-kind
ledger entries in one batched retain + sync (a mass-recovery first
sweep must not turn into thousands of full-table ledger clones on the
scan task), with nil notice UUIDs mapping to None per the repo-wide
defensive-UUID invariant so unversioned entries match unversioned
notices only. Notices are best-effort by design — a lost or capped-out
notice leaves the entry to expire through its own attempts/age limits,
because the ledger is a retry oracle, not a source of truth; other
buckets' notices stay queued for their own scanners. Neither persistent
format changes; old nodes that keep double-booking remain harmless.

Co-authored-by: heihutu <[email protected]>
2026-08-19 17:31:26 +00:00
housemeandGitHub 0126f359e3 fix(heal): stop O(window) memmove in the heal result window (#6272) 2026-08-19 23:33:33 +08:00
be7f684718 refactor(heal): remove the dead MRF heal-type path (#6275)
HealType::MRF (a #1664-era "metadata repair file" task kind) had no
production construction site left: its only builder lived in the
HealEvent -> HealRequest converter, and the HealEvent/HealEventHandler
queue itself had zero production references — both were superseded by
the MrfIntent pipeline (mrf_queue.rs), which produces Object/Metadata/
ECDecode requests and never an MRF task. The dead path nevertheless
carried ~700 lines: the whole event.rs module, the heal_mrf executor,
a dedup-key arm, an overlap arm with the "\u{0}mrf" sentinel bucket
hack, per-kind labels, and an empty MrfRuntime::record_accept shell.

Deleting the variant is compile-time safe: HealType has no Serialize
derive, the protos wire enums carry no heal-type discriminant (the
receiver rebuilds it from HealChannelRequest fields), the MRF journal
encodes MrfKind (1/2/3), and the scanner pending-heal ledger uses its
own kind enum — none of them can name an MRF task.

Also resolves the in-crate naming clash where "MRF" denoted both the
dead task kind and the live mission-repair-feed loop; the loop stays,
the task kind goes.

Co-authored-by: heihutu <[email protected]>
2026-08-19 14:52:36 +00:00
housemeandGitHub d404e1bb8a refactor(heal,scanner): clean up dead heal/scanner code, flags, and metrics (HS-09/10/19/20) (#6256) 2026-08-19 13:11:46 +08:00
Zhengchao AnandGitHub 1741f79d7d test(audit,heal): assert three more smoke tests (#6248)
`audit_runtime_facade_stops_empty_replay_workers` called the stop path and checked nothing, the same shape as the notify facade test in the previous commit. It now asserts the worker manager is empty afterwards and that a second call — which shutdown paths make — stays harmless.

The two heal timestamp tests bound their fields to `_`. Both timestamps come from `SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default()`, so a pre-epoch clock yields 0; binding to `_` could not tell that apart from a real reading, which is precisely what the tests said they were guarding. They now require the value to be past 2020-01-01, and `last_update` not to predate `start_time`.

`test_config_parsing_with_multiple_instances` is left as it was — see the issue comment. Asserting on it turned up something bigger than a missing assertion.

Refs backlog#1836
2026-08-19 12:20:54 +08:00
c7c5a8df6a test(heal): cover privileged mount readiness (#6231)
Add Linux-only ignored replacement readiness tests for independent mount admission and same-device sibling rejection.

Co-authored-by: heihutu <[email protected]>
2026-08-19 01:27:55 +00:00
a5800033bd feat(heal): incremental status cursors and typed overlap policy (HS-06) (#6206)
* feat(heal): incremental heal status cursors and typed overlap policy (HS-06)

Incremental results: every retained result item now carries a monotonic
sequence number. The status query accepts a client cursor (sinceSeq on
the admin wire, Option<u64> internally) and returns only newer items,
plus nextSeq (the next cursor) and minSeq (the oldest retained
sequence). A cursor that fell behind the 1024-item retention window is
flagged through the existing truncated signal together with minSeq so
the client can restart from it. Sequencing survives task completion:
the completion archive stores the seq-stamped window. None keeps the
exact legacy full-snapshot behavior, so existing clients see no change.

Typed overlap handling for admin starts: RUSTFS_HEAL_OVERLAP_POLICY
(merge default | minio_error). Under minio_error, an admin start whose
path overlaps an active or queued task rejects with typed
already-running / overlapping-paths admission reasons (surfaced through
reason_label in the admin error body, sharing the existing
OperationAborted site because the s3s footprint ratchet forbids new
s3_error! sites); an exact duplicate start rejects with
already-running instead of silently merging. Scanner/autoheal/
read-repair sources never take the rejection path.

forceStart semantics now match MinIO for admin requests: an admin
forceStart first cancels the overlapping active admin task, then
admits the replacement.

Wire: the heal-control Query command grows an optional sinceSeq
(defaulted and skipped when absent, so older peers stay compatible);
the admin handler accepts the sinceSeq query parameter; the local
channel query gains the same cursor.

Tests: seq monotonicity and incremental slicing, window slide moving
minSeq with lagging-cursor flags, overlap matrix (same/containing/
contained/disjoint x policy x source), forceStart cancel-then-admit,
and the completion-archive window handoff.

Co-Authored-By: heihutu <[email protected]>

* style: fmt after main merge

---------

Co-authored-by: heihutu <[email protected]>
Co-authored-by: zhi22915 <[email protected]>
2026-08-18 16:09:30 +08:00
a08de9229b feat(heal): wire MRF intents with durable repair journal (HS-01) (#6189)
* feat(common): add MRF intent channel and Mrf request source (HS-01)

Introduce the producer-facing half of the mission repair feed: a global
bounded (8192) channel carrying lightweight MrfIntent values from IO
error paths, plus the RUSTFS_HEAL_MRF_ENABLE delivery kill-switch and
config constants for queue/journal sizing. Delivery is strictly
non-blocking (try_send, drop-on-full) so it can sit on decode-failure
and partial-write paths without adding latency. HealRequestSource grows
a 'mrf' variant so admission accounting can attribute replayed intents.

Part of backlog#1865 (option a: wire HealEvent-style intents with a
durable retry ledger).

Co-Authored-By: heihutu <[email protected]>

* feat(heal): add MRF queue, durable journal, and intent consumer (HS-01)

Consumer half of the mission repair feed: a bounded pending queue
(100k intents / 8 MiB dual ceiling, drop-newest on overflow), a durable
journal at buckets/.heal/mrf/journal.bin holding the unaccepted pending
snapshot, and a consumer task that batches intents off the global
channel, translates them into prioritized heal requests (decode
failure -> Urgent ECDecode, metadata corruption -> High Metadata,
partial write -> Normal object heal), and retries full admissions with
a 5s backoff and a 3-attempt ceiling.

Durability: every journal record carries its own CRC32 and a
format/version header, so a torn tail truncates cleanly at replay; the
journal is deleted after a successful replay and when the pending set
drains (mirroring MinIO's post-replay list.bin unlink). Losing the last
500 ms flush window is acceptable: replayed duplicates merge via the
manager dedup key and read-repair remains the safety net.

Metrics: rustfs_heal_mrf_queue_depth/_queue_bytes, _dropped_total
{reason}, _replayed_total, _journal_bytes, _journal_fsync_total.
The consumer is wired at heal runtime bootstrap right after manager
start, honoring RUSTFS_HEAL_MRF_ENABLE (default on, rollback = off).

Tests: unit tests for the dual ceiling, record roundtrip, torn-tail
truncation, and the priority mapping; integration tests against a real
4-disk ECStore proving channel intents reach the manager queue as
Urgent/mrf-attributed requests and journal replay arms intents, drops
torn tails, and removes the file.

Part of backlog#1865 (option a).

Co-Authored-By: heihutu <[email protected]>

* feat(ecstore,scanner): deliver MRF intents from error paths (HS-01)

Wire the three production delivery points, each a single non-blocking
try_send next to the existing in-memory heal paths, which stay as the
fast path:

- read.rs decode-error branch: DecodeFailure intent beside the existing
  read-repair submit, so an Urgent ECDecode request survives restarts
  even when the Low-priority read-repair request was dropped or lost.
- add_partial: PartialWrite intent, giving partial-write recovery a
  durable Normal-priority object heal across restarts.
- scanner_folder metadata-corruption classification: MetadataCorruption
  intent beside the existing High-priority scanner heal request.

All three are on error paths only: zero cost on healthy IO.

Part of backlog#1865 (option a).

Co-Authored-By: heihutu <[email protected]>

* fix: include mrf heal source counts

Co-Authored-By: heihutu <[email protected]>

* fix: keep node heal status wire compatibility

Co-Authored-By: heihutu <[email protected]>

---------

Co-authored-by: heihutu <[email protected]>
2026-08-18 12:43:27 +08:00
360bceafce feat(heal): add progress and trace observability (#6179)
* feat(heal): track erasure set progress baseline

Record erasure-set heal byte progress from per-object results and seed progress totals from complete usage-cache snapshots when available.

Keep usage-cache failures observational so heal execution continues without a baseline.

Co-Authored-By: heihutu <[email protected]>

* feat(heal): skip filtered erasure set versions

Skip erasure-set versions written after the durable heal start time, and queue lifecycle-expired versions for expiry before skipping them.

Track new-version and ILM-expired skips separately so progress can explain completed baseline work without treating these skips as retry-blocking failures.

Co-Authored-By: heihutu <[email protected]>

* feat(heal): wire abandoned data-dir cleanup check

Connect check_abandoned_parts through ECStore, pool, and set layers so heal can invoke the existing orphan data-dir reclaim path instead of returning NotImplemented.

Add dry-run support to the reclaim scan and cover dry-run plus scoped set behavior with regression tests.

Co-Authored-By: heihutu <[email protected]>

* feat(obs): add heal scanner trace bus

Introduce an in-process broadcast trace bus with typed heal and scanner events, lazy event construction, and bounded lagged-subscriber behavior.

Cover zero-subscriber publishing, subscription delivery, drop accounting, and lagged receivers with focused common-crate tests.

Co-Authored-By: heihutu <[email protected]>

* feat(obs): stream heal trace events from admin API

Wire the admin trace endpoint to the common trace bus for heal/scanner events, including kind, regex, and threshold filtering.

Co-Authored-By: heihutu <[email protected]>

* feat(obs): emit heal trace events

Publish heal task lifecycle and abandoned-parts cleanup events through the common trace bus so the admin trace stream has live heal diagnostics.

Co-Authored-By: heihutu <[email protected]>

* feat(obs): emit scanner trace events

Publish scanner folder, lifecycle action, and heal-candidate events through the common trace bus for live admin scanner diagnostics.

Co-Authored-By: heihutu <[email protected]>

* fix(heal): route data usage loader through storage api

Keep ECStore data-usage facade access behind the heal storage_api boundary so architecture migration guards can validate the heal progress path.

Co-Authored-By: heihutu <[email protected]>

* perf(heal): avoid lifecycle snapshots on ordinary heal pages

Only request lifecycle object snapshots when the heal pass has lifecycle expiry context. This keeps ordinary listing and disk-walk pages from cloning FileInfo/ObjectInfo payloads while preserving the skip path that queues expired versions.

Co-Authored-By: heihutu <[email protected]>

* test(heal): update bug-fix mocks for lifecycle snapshots

Carry the lifecycle snapshot opt-in argument through the remaining heal bug-fix test mocks so all-targets clippy covers the updated storage trait.

Co-Authored-By: heihutu <[email protected]>

* test(rustfs): sync heal storage mock signature

Update the rustfs storage RPC test mock for the lifecycle snapshot opt-in argument and cover it with rustfs all-targets clippy.

Co-Authored-By: heihutu <[email protected]>

* test(e2e): allocate smoke ports across nextest processes

Serialize E2E port selection with a small /tmp allocator so nextest workers do not reuse the same just-released ephemeral port before RustFS binds it.

Co-Authored-By: heihutu <[email protected]>

---------

Co-authored-by: heihutu <[email protected]>
2026-08-18 08:29:29 +08:00
8d3511c1b3 fix(heal): preserve timeout budget across retries (#6101)
Co-authored-by: Henry Guo <[email protected]>
2026-08-16 10:28:11 +08:00
161e515c72 chore(heal): remove seven dead error variants (#6031)
heal::Error carried six variants with zero construction and zero match sites (ConfigurationError, NotFound, TaskAlreadyExists, ManagerNotRunning, EventProcessingFailed, ProgressTrackingFailed) plus IO(String), which was never constructed either — its only appearances were two or-pattern match arms that could never fire (task.rs's demotion match and the recoverability classifier). All seven are deleted and the two or-patterns lose their dead alternative.

Config(String) stays (live, four construction sites); Io(std::io::Error) stays; the retry classifier's behavior is untouched per the issue constraint — removing an arm that can never match is not a classification change.

Ref rustfs/backlog#1831 (PR3).

Co-authored-by: cxymds <[email protected]>
Co-authored-by: houseme <[email protected]>
2026-08-14 08:18:47 +08:00
aa4d3317ed perf(ecstore): guard inline data-read metadata early-stop (#6069)
Add a default-off inline-only data-read metadata early-stop gate that verifies inline plaintext before cancelling pending metadata tasks.

Keep non-inline, prepared, and request-shape-sensitive reads on full fanout, and record scheduled/completed/cancelled ReadVersion lifecycle metrics for normal fanout completion.

Co-authored-by: heihutu <[email protected]>
2026-08-13 21:40:14 +08:00
Zhengchao AnandGitHub 0a246e3736 test: assert real behavior in three assertion-less tests (#5993)
test_format_v1 (ecstore layout::format) only printed its results; the pinned v1 format.json literal never parsed at all because "this": null fails Uuid deserialization, and the Err was silently discarded. Fix the fixture to the real on-disk shape (MinIO and RustFS always write a concrete disk UUID there) and assert a serialize->parse roundtrip identity plus every pinned field of the literal.

test_console_cors_configuration discarded all four parse_cors_origins results; parse_cors_origins returns an opaque CorsLayer, so the test now drives real CORS preflight requests through an axum router and asserts the allow-origin outcomes: wildcard answers any origin with *, a configured list echoes listed origins and refuses unlisted ones, empty/unset configurations allow no cross-origin caller.

test_heal_channel_processor_new only constructed the processor; it now asserts the response channel accepts a send.

Ref rustfs/backlog#1836 (PR1).
2026-08-12 14:37:01 +00:00
7a4a3d27c6 fix(heal): cancel cluster tasks from root stop (#5978)
Co-authored-by: Henry Guo <[email protected]>
2026-08-12 20:46:35 +08:00
bc888931fd fix(heal): quiet deferred replacement recovery logs (#5954)
Treat recovery-directory lookup on a replacement endpoint already deferred as replacement_path_unavailable as an expected debug diagnostic instead of a durable generation conflict.

Keep real survivor recovery conflicts and corrupt records on the existing warning/blocking path.

Co-authored-by: heihutu <[email protected]>
2026-08-11 09:35:06 +00:00
f1ff9a36bc test(heal): cover replacement terminal recovery (#5920)
Co-authored-by: heihutu <[email protected]>
2026-08-10 14:51:26 +08:00
276eea1fba test(heal): cover replacement target evidence failures (#5919)
Co-authored-by: heihutu <[email protected]>
2026-08-10 14:50:55 +08:00
f17ea7f146 fix(heal): harden replacement rebuild tracking (#5892)
* fix(heal): gate auto replacement formatting

Co-Authored-By: heihutu <[email protected]>

* fix(heal): require replacement target outcomes

Co-Authored-By: heihutu <[email protected]>

* fix(heal): bind resumes to replacement targets

Co-Authored-By: heihutu <[email protected]>

* fix(heal): fence healing marker ownership

Co-Authored-By: heihutu <[email protected]>

* test(heal): cover replacement target completion

Co-Authored-By: heihutu <[email protected]>

* docs(heal): clarify replacement recovery status

Co-Authored-By: heihutu <[email protected]>

* fix(heal): canonicalize replacement target checks

Co-Authored-By: heihutu <[email protected]>

* fix(heal): satisfy marker test module lint

Co-Authored-By: heihutu <[email protected]>

* fix(heal): scope automatic replacement format

Co-Authored-By: heihutu <[email protected]>

* fix(heal): require a mounted replacement target

Co-Authored-By: heihutu <[email protected]>

* fix(ecstore): avoid cloned ref slice in test

Co-Authored-By: heihutu <[email protected]>

* fix(heal): revalidate replacement before scanning

Co-Authored-By: heihutu <[email protected]>

* fix(heal): reset stale resume checkpoints

Co-Authored-By: heihutu <[email protected]>

* fix(heal): release scanner disk map before probing

Co-Authored-By: heihutu <[email protected]>

* fix(heal): persist replacement intent before format

Co-Authored-By: heihutu <[email protected]>

* fix(heal): fail closed on mountinfo read errors

Co-Authored-By: heihutu <[email protected]>

* fix(heal): fence replacement target identity

Co-Authored-By: heihutu <[email protected]>

* fix(heal): order replacement completion cleanup

Co-Authored-By: heihutu <[email protected]>

* fix(heal): atomically seal replacement completion

Co-Authored-By: heihutu <[email protected]>

* test(heal): census replacement target shards

Co-Authored-By: heihutu <[email protected]>

* fix(heal): fence replacement recovery ownership

Co-Authored-By: heihutu <[email protected]>

* fix(heal): preserve replacement recovery anchors

Co-Authored-By: heihutu <[email protected]>

* fix(heal): satisfy replacement recovery lint gates

Co-Authored-By: heihutu <[email protected]>

* fix(ecstore): bind replacement identity to mount lease

Co-Authored-By: heihutu <[email protected]>

* test(heal): cover durable replacement recovery states

Co-Authored-By: heihutu <[email protected]>

* fix(heal): validate persisted resume task identifiers

Co-Authored-By: heihutu <[email protected]>

* fix(ecstore): avoid blocking replacement marker CAS

Co-Authored-By: heihutu <[email protected]>

* fix(heal): report failed marker rollback

Co-Authored-By: heihutu <[email protected]>

* test(heal): pin replacement resume schema compatibility

Co-Authored-By: heihutu <[email protected]>

* fix(heal): preserve durable recovery anchors

Co-Authored-By: heihutu <[email protected]>

* fix(ecstore): preserve public disk path semantics

Co-Authored-By: heihutu <[email protected]>

* test(heal): use canonical replacement task ids

Co-Authored-By: heihutu <[email protected]>

* test(heal): cover automatic replacement in 3x4 cluster

Co-Authored-By: heihutu <[email protected]>

* fix(heal): verify replacement target commits

Co-Authored-By: heihutu <[email protected]>

* fix(heal): persist replacement completion proof

Co-Authored-By: heihutu <[email protected]>

* feat(heal): expose durable replacement status

Co-Authored-By: heihutu <[email protected]>

* fix(heal): bound durable replacement discovery

Co-Authored-By: heihutu <[email protected]>

* fix(heal): remove replacement readiness bypass

Co-Authored-By: heihutu <[email protected]>

* fix(heal): retry terminal replacement cleanup

Co-Authored-By: heihutu <[email protected]>

* fix(heal): isolate replacement intents from legacy resume

Co-Authored-By: heihutu <[email protected]>

* fix(heal): migrate legacy replacement intents at startup

Co-Authored-By: heihutu <[email protected]>

* style(heal): apply strict clippy fix

Co-Authored-By: heihutu <[email protected]>

* fix(heal): prioritize active replacement recovery state

Co-Authored-By: heihutu <[email protected]>

* fix(heal): bind readiness to the admitted mount lease

Co-Authored-By: heihutu <[email protected]>

* fix(heal): atomically publish replacement intents

Co-Authored-By: heihutu <[email protected]>

* fix(heal): isolate replacement recovery directory

Co-Authored-By: heihutu <[email protected]>

* fix(heal): tolerate an empty recovery directory

Co-Authored-By: heihutu <[email protected]>

* style(heal): remove redundant disk bytes conversion

Co-Authored-By: heihutu <[email protected]>

* fix(heal): reconcile proof-first replacement recovery

Co-Authored-By: heihutu <[email protected]>

* fix(heal): fence torn intent recovery

Co-Authored-By: heihutu <[email protected]>

* test(heal): cover replacement migration conflicts

Co-Authored-By: heihutu <[email protected]>

* fix(ecstore): fence replacement lease mount identity

Co-Authored-By: heihutu <[email protected]>

* test(heal): cover missing replacement path admission

Co-Authored-By: heihutu <[email protected]>

* fix(heal): reject conflicting legacy completion proof

Co-Authored-By: heihutu <[email protected]>

* fix(ecstore): fall back to proc mount identity

Co-Authored-By: heihutu <[email protected]>

* feat(admin): expose replacement recovery status

Surface the local durable replacement recovery snapshot in the background heal status response so operators can tell whether replacement cleanup is definitive or still pending.

Co-Authored-By: heihutu <[email protected]>

* fix(heal): keep replacement status compatible

Keep the existing background heal status response wire-compatible while retaining the Linux mount lease cleanup needed for the replacement recovery branch.

Co-Authored-By: heihutu <[email protected]>

* style(ecstore): match linux mount lease formatting

Keep Linux rustfmt output stable for the replacement mount lease comparison.

Co-Authored-By: heihutu <[email protected]>

* fix(ecstore): qualify mount lease test constant

Use the disk module path for the format config constant in the Linux mount lease regression test.

Co-Authored-By: heihutu <[email protected]>

* fix(ecstore): keep procfd mount roots directory-safe

Use a procfd path with an explicit directory component so Unix directory guards can open the replacement mount lease root with O_NOFOLLOW while preserving handle-relative I/O semantics.

Co-Authored-By: heihutu <[email protected]>

* fix(ecstore): delete empty leased buckets via dirfd

Use the held mount lease fd as the parent for non-force empty bucket deletion on Linux so procfd-rooted paths do not get rejected as BucketNotEmpty. Also make the download-part OpenOptions truncate behavior explicit and keep fsync test recording stable across procfd canonicalization.

Co-Authored-By: heihutu <[email protected]>

* fix(ecstore): scan leased bucket paths for emptiness

Use the local disk I/O root for bucket emptiness probes before non-force bucket deletion and table-bucket metadata checks. This keeps validation on the same mount instance as the subsequent local disk delete path.

Co-Authored-By: heihutu <[email protected]>

* test(ecstore): align lease path test probes

Co-Authored-By: heihutu <[email protected]>

* fix(heal): block unsafe replacement recovery restarts

Co-Authored-By: heihutu <[email protected]>

* fix(heal): defer blocked replacement candidates

Co-Authored-By: heihutu <[email protected]>

* fix(heal): retry transient replacement discovery

Co-Authored-By: heihutu <[email protected]>

* fix(heal): keep transient recovery errors retryable

Co-Authored-By: heihutu <[email protected]>

* fix(heal): block corrupt legacy replacement state

Co-Authored-By: heihutu <[email protected]>

* fix(heal): classify flat replacement intent corruption

Co-Authored-By: heihutu <[email protected]>

* fix(heal): keep transient resume loads retryable

Classify malformed legacy replacement state as blocking corruption while preserving disk and transient load failures for retry. This avoids permanently blocking replacement recovery on temporary storage errors.

Co-Authored-By: heihutu <[email protected]>

* fix(heal): avoid latching transient legacy publishes

Co-Authored-By: heihutu <[email protected]>

* fix(heal): retry blocked legacy migrations

Co-Authored-By: heihutu <[email protected]>

* fix(heal): defer blocked startup recoveries

Co-Authored-By: heihutu <[email protected]>

* fix(ecstore): preserve disk sync limiter across lease roots

Co-Authored-By: heihutu <[email protected]>

---------

Co-authored-by: heihutu <[email protected]>
Co-authored-by: zhi22915 <[email protected]>
2026-08-10 08:32:47 +08:00
terem42andGitHub 578d02977e fix(heal): log the number of drives actually healed, not the drives consulted (#5871) 2026-08-09 14:10:11 +08:00
cxymdsandGitHub 47369ff027 fix(heal): defer scoped repair on suspended pools (#5876) 2026-08-09 11:50:17 +08:00
cxymdsandGitHub b4133d69e6 fix(heal): respect scoped object repair limits (#5855) 2026-08-09 07:23:16 +08:00
cxymdsandGitHub 3bad829b9a test(heal): cover partial rename retry admission (#5772) 2026-08-06 22:13:29 +08:00
Zhengchao AnandGitHub d2e5346044 fix(heal): demote per-object heal logs and cap erasure-set failure warns (#5727)
fix(heal): demote per-object logs and cap erasure-set failure warns

Follow-up to rustfs/rustfs#5716. Per-object heal task kinds (Object/Metadata/MRF/ECDecode) queued by MRF/autoheal/scanner loops emitted info!/warn!/error! lines per object: task lifecycle (started/completed/timed_out/failed), the missing-object warn, queue admission full/drop/displacement warns, retry-admission decisions, and uncapped per-object warns in erasure-set sweeps.

Add a shared demote_to_debug_when! macro that keeps aggregate task kinds and admin/internal requests at operator-visible levels while demoting per-object occurrences to debug!, sample-cap the erasure-set transient_skip/failed warns per bucket via take_failure_log_sample (reusing the heal_bucket_objects precedent), demote the per-retry admission decision logs to debug! (covered by rustfs_heal_admission_total and the scheduler task_retrying/task_failed events), and record the previously unmetered duplicate-admission outcome.

Extend scripts/check_logging_guardrails.sh with injection-verified regression guards and run it in both quick-checks jobs (ci.yml and its ci-docs-only.yml mirror).
2026-08-05 02:43:57 +00:00
cxymdsandGitHub 31959b90db fix(heal): harden resumable set repair failures (#5693)
* fix(heal): enforce resumable task control

* fix(ecstore): surface bucket and metadata heal errors

* chore: refresh guardrail path references

---------

Signed-off-by: cxymds <[email protected]>
2026-08-04 12:35:58 +00:00
78d6918c52 feat: extend hotpath coverage across crates (#5505)
Add opt-in hotpath feature surfaces to every workspace crate and wire the root rustfs feature passthrough for function, allocation, and CPU profiling.

Add a focused set of function-level measurements for scanner, heal, lock, target replay, IAM, KMS, Keystone, trusted proxy, and capacity paths without adding request-scoped primitive wrappers.

Co-authored-by: heihutu <[email protected]>
2026-07-31 05:42:19 +00:00
Zhengchao AnandGitHub e154e0e4a2 chore(heal): enforce workspace lint policy (#5426) 2026-07-29 10:00:51 +00:00
Zhengchao AnandGitHub d42bc52f8b test(heal): avoid live disk removal race (#5421) 2026-07-29 17:29:18 +08:00
cxymdsandGitHub d7afa4e38e fix(heal): report partial rename failures (#5355)
* fix(heal): report partial rename failures

* fix(log-analyzer): track partial heal rename failures
2026-07-28 17:37:02 +08:00
cxymdsandGitHub fe67af3524 fix(heal): coordinate cluster-wide control operations (#5003) 2026-07-20 09:40:46 +00:00
4290f390dd fix(heal): aggregate status across cluster nodes (#4990)
* fix(rpc): bind internode auth to exact targets

* fix(heal): initialize the runtime atomically

* fix(heal): aggregate status across cluster nodes

---------

Co-authored-by: Zhengchao An <[email protected]>
2026-07-19 15:25:52 +00:00
cxymdsandGitHub 1ac0841f6f fix(heal): initialize the runtime atomically (#4989)
* fix(rpc): bind internode auth to exact targets

* fix(heal): initialize the runtime atomically
2026-07-19 14:20:37 +00:00
f3a7a4b0da chore(deps): localize workspace dependency features (#4888)
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root.

Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit.

Co-authored-by: heihutu <[email protected]>
2026-07-16 03:55:27 +00:00
housemeandGitHub 5ef2731a6b ci: tune build workflow runners (#4871)
* chore: extend build timeout and trim dev deps

* ci: switch linux builds to ubicloud runners

* ci: use larger linux build runners
2026-07-15 20:58:37 +08:00
28ee1e5e1a test(heal): tolerate by-design "retry scheduled" deferral in resume e2e (#4867)
`test_heal_resume_across_page_boundary_e2e` panicked whenever the
erasure-set healer deferred a single object version to a later heal
cycle. Under load a per-version `heal_object` can hit a transient error;
`ErasureSetHealer::heal_bucket_with_resume` then persists its
resume/checkpoint state and returns a terminal `Failed { .. "retry
scheduled" }`, expecting a fresh heal run to finish the job. In
production the background scanner is that next run — the e2e had no such
follow-up, so the first task's `Failed` state failed the test. This is a
pre-existing rare flake (the wiped-disk heal is otherwise correct); it is
unrelated to any policy/proptest work that happened to surface it in CI.

Drive the heal to a genuine `Completed` instead: on a `Failed` carrying
the `retry scheduled` marker (retry budget still remaining) re-submit the
idempotent heal (`force_start`), mirroring the production scanner, up to a
small bound. A `Failed` without that marker (e.g. `exhausted retries`) or
any other non-`Completed` terminal state still fails the test, and the
strict per-version data-restoration assertions still run only after a
real `Completed`. `wait_for_task` is refactored onto the shared
`await_terminal_status` poller; its panic-on-failure semantics for the
unversioned-bucket heal are unchanged.

Test-only change; no production heal code is touched.

Co-authored-by: heihutu <[email protected]>
2026-07-15 19:53:35 +08:00
Zhengchao AnandGitHub f05a69d51b test(utils): add rustfs-test-utils crate and shared ECStore bootstrap (#4850)
* test(utils): add rustfs-test-utils crate, absorb heal/iam ECStore bootstrap

backlog#1153 infra-1. The ~50-line "build a real temp-disk ECStore"
bootstrap was copy-pasted (and drifting) across the heal and iam
integration tests. This adds crates/test-utils (rustfs-test-utils, a
dev-dependency-only crate) owning that bootstrap and converts the four
copies into thin wrappers:

- TestECStoreEnvBuilder: disk_count (default 4), prefix (uuid-suffixed
  /tmp dir), base_dir (caller-owned dir, e.g. tempfile::TempDir),
  init_bucket_metadata (default true; the iam bootstrap test opts out
  to preserve its historical semantics). TestECStoreEnv exposes
  temp_root/disk_paths/ecstore plus a versioned-bucket helper, and
  init_tracing() replaces the per-file Once blocks.
- All rustfs_ecstore imports stay behind src/ecstore_test_compat.rs,
  the sanctioned test-compat boundary pattern (mirrors
  crates/iam/tests/ecstore_test_compat).
- heal: heal_integration_test / heal_b5_versioned_regression_test /
  heal_b920_subquorum_union_test drop their setup_test_env{,_n} copies
  for heal_env{,_n} wrappers; the tests/storage_api.rs integration
  surface shrinks to what test bodies still touch.
- iam: iam_bootstrap_no_lock_test drops build_local_ecstore; its
  ecstore_test_compat fixture shrinks to SetupType +
  update_erasure_type.

rg 'async fn setup_test_env' crates/heal crates/iam now returns 0.
Scanner's lifecycle tests are deliberately NOT absorbed (gated on
ilm-1; 14 of 15 are #[ignore]d today). Net -230 lines.

* fix(heal): drop tokio::fs import orphaned by the b920 bootstrap move

* fix(heal): drop tokio::fs import orphaned by the b5 bootstrap move
2026-07-15 16:08:30 +08:00
cxymdsandGitHub 23fa008ed8 fix(heal): retry failed objects within bucket scans (#4834) 2026-07-15 14:32:07 +08:00
Zhengchao AnandGitHub 535d672b1f fix(admin): report heal runtime state (#4786) 2026-07-13 11:02:13 +08:00
63b4568f85 fix(ecstore): reclaim orphan data dirs on the healthy heal path (#4781)
PR #4356 wired `reclaim_orphan_data_dirs` only into `heal_object`'s
post-heal tail, which runs after the `disks_to_heal_count == 0` early
return. That early return is exactly the state of the objects the sweep
targets: a valid `xl.meta` with all shards present plus a leaked
pre-#3510 data dir needs no shard healing, so a healthy heal returned
before reclaim and swept nothing. On a healthy deployment (single node,
no degraded disks) the reclaim was therefore dead code — an admin heal
walked the objects, "healed" them, and reclaimed no leaked space.

Run the best-effort reclaim on the `disks_to_heal_count == 0` path as
well, gated on `!opts.dry_run`. The shared match+log block is factored
into `reclaim_orphan_data_dirs_best_effort` so both exits behave
identically. A reclaim failure still never fails the heal.

Adds an end-to-end regression: put a healthy non-inline object, plant an
unreferenced UUID data dir under it on every disk that holds the object,
then drive `heal_object`. A dry-run heal must leave the stray in place; a
real heal must reclaim it while preserving the live data dirs, `xl.meta`,
and object contents. The test fails against the pre-fix control flow.

Refs #3231, #3191, #4356.

Co-authored-by: heihutu <[email protected]>
2026-07-12 14:19:47 +00:00
13bdca6762 build(toolchain): switch Rust channel to stable (#4775)
* Change Rust toolchain channel to stable

Signed-off-by: houseme <[email protected]>

* style: apply clippy --fix and cargo fix lint suggestions

Run `cargo clippy --fix --all-targets --all-features` and
`cargo fix --lib --all-targets` across the workspace, then resolve the
remaining warnings by hand:

- collapse needless borrows in `format!` args, prefer `?` over explicit
  early returns, and use `.values()` / `.flatten()` iterator adapters
- rewrite the `Md5` scan loop via `manual_flatten` and re-indent the
  `select!` macro body (rustfmt skips macro interiors)
- annotate the intentional dead-code `Md5` inherent methods (constructed
  only by the test factory) with `#[allow(dead_code)]`

Behavior is unchanged.

Co-Authored-By: heihutu <[email protected]>

---------

Signed-off-by: houseme <[email protected]>
Co-authored-by: heihutu <[email protected]>
2026-07-12 18:59:43 +08:00