fix(heal): harden replacement rebuild tracking (#5892)
* fix(heal): gate auto replacement formatting Co-Authored-By: heihutu <[email protected]> * fix(heal): require replacement target outcomes Co-Authored-By: heihutu <[email protected]> * fix(heal): bind resumes to replacement targets Co-Authored-By: heihutu <[email protected]> * fix(heal): fence healing marker ownership Co-Authored-By: heihutu <[email protected]> * test(heal): cover replacement target completion Co-Authored-By: heihutu <[email protected]> * docs(heal): clarify replacement recovery status Co-Authored-By: heihutu <[email protected]> * fix(heal): canonicalize replacement target checks Co-Authored-By: heihutu <[email protected]> * fix(heal): satisfy marker test module lint Co-Authored-By: heihutu <[email protected]> * fix(heal): scope automatic replacement format Co-Authored-By: heihutu <[email protected]> * fix(heal): require a mounted replacement target Co-Authored-By: heihutu <[email protected]> * fix(ecstore): avoid cloned ref slice in test Co-Authored-By: heihutu <[email protected]> * fix(heal): revalidate replacement before scanning Co-Authored-By: heihutu <[email protected]> * fix(heal): reset stale resume checkpoints Co-Authored-By: heihutu <[email protected]> * fix(heal): release scanner disk map before probing Co-Authored-By: heihutu <[email protected]> * fix(heal): persist replacement intent before format Co-Authored-By: heihutu <[email protected]> * fix(heal): fail closed on mountinfo read errors Co-Authored-By: heihutu <[email protected]> * fix(heal): fence replacement target identity Co-Authored-By: heihutu <[email protected]> * fix(heal): order replacement completion cleanup Co-Authored-By: heihutu <[email protected]> * fix(heal): atomically seal replacement completion Co-Authored-By: heihutu <[email protected]> * test(heal): census replacement target shards Co-Authored-By: heihutu <[email protected]> * fix(heal): fence replacement recovery ownership Co-Authored-By: heihutu <[email protected]> * fix(heal): preserve replacement recovery anchors Co-Authored-By: heihutu <[email protected]> * fix(heal): satisfy replacement recovery lint gates Co-Authored-By: heihutu <[email protected]> * fix(ecstore): bind replacement identity to mount lease Co-Authored-By: heihutu <[email protected]> * test(heal): cover durable replacement recovery states Co-Authored-By: heihutu <[email protected]> * fix(heal): validate persisted resume task identifiers Co-Authored-By: heihutu <[email protected]> * fix(ecstore): avoid blocking replacement marker CAS Co-Authored-By: heihutu <[email protected]> * fix(heal): report failed marker rollback Co-Authored-By: heihutu <[email protected]> * test(heal): pin replacement resume schema compatibility Co-Authored-By: heihutu <[email protected]> * fix(heal): preserve durable recovery anchors Co-Authored-By: heihutu <[email protected]> * fix(ecstore): preserve public disk path semantics Co-Authored-By: heihutu <[email protected]> * test(heal): use canonical replacement task ids Co-Authored-By: heihutu <[email protected]> * test(heal): cover automatic replacement in 3x4 cluster Co-Authored-By: heihutu <[email protected]> * fix(heal): verify replacement target commits Co-Authored-By: heihutu <[email protected]> * fix(heal): persist replacement completion proof Co-Authored-By: heihutu <[email protected]> * feat(heal): expose durable replacement status Co-Authored-By: heihutu <[email protected]> * fix(heal): bound durable replacement discovery Co-Authored-By: heihutu <[email protected]> * fix(heal): remove replacement readiness bypass Co-Authored-By: heihutu <[email protected]> * fix(heal): retry terminal replacement cleanup Co-Authored-By: heihutu <[email protected]> * fix(heal): isolate replacement intents from legacy resume Co-Authored-By: heihutu <[email protected]> * fix(heal): migrate legacy replacement intents at startup Co-Authored-By: heihutu <[email protected]> * style(heal): apply strict clippy fix Co-Authored-By: heihutu <[email protected]> * fix(heal): prioritize active replacement recovery state Co-Authored-By: heihutu <[email protected]> * fix(heal): bind readiness to the admitted mount lease Co-Authored-By: heihutu <[email protected]> * fix(heal): atomically publish replacement intents Co-Authored-By: heihutu <[email protected]> * fix(heal): isolate replacement recovery directory Co-Authored-By: heihutu <[email protected]> * fix(heal): tolerate an empty recovery directory Co-Authored-By: heihutu <[email protected]> * style(heal): remove redundant disk bytes conversion Co-Authored-By: heihutu <[email protected]> * fix(heal): reconcile proof-first replacement recovery Co-Authored-By: heihutu <[email protected]> * fix(heal): fence torn intent recovery Co-Authored-By: heihutu <[email protected]> * test(heal): cover replacement migration conflicts Co-Authored-By: heihutu <[email protected]> * fix(ecstore): fence replacement lease mount identity Co-Authored-By: heihutu <[email protected]> * test(heal): cover missing replacement path admission Co-Authored-By: heihutu <[email protected]> * fix(heal): reject conflicting legacy completion proof Co-Authored-By: heihutu <[email protected]> * fix(ecstore): fall back to proc mount identity Co-Authored-By: heihutu <[email protected]> * feat(admin): expose replacement recovery status Surface the local durable replacement recovery snapshot in the background heal status response so operators can tell whether replacement cleanup is definitive or still pending. Co-Authored-By: heihutu <[email protected]> * fix(heal): keep replacement status compatible Keep the existing background heal status response wire-compatible while retaining the Linux mount lease cleanup needed for the replacement recovery branch. Co-Authored-By: heihutu <[email protected]> * style(ecstore): match linux mount lease formatting Keep Linux rustfmt output stable for the replacement mount lease comparison. Co-Authored-By: heihutu <[email protected]> * fix(ecstore): qualify mount lease test constant Use the disk module path for the format config constant in the Linux mount lease regression test. Co-Authored-By: heihutu <[email protected]> * fix(ecstore): keep procfd mount roots directory-safe Use a procfd path with an explicit directory component so Unix directory guards can open the replacement mount lease root with O_NOFOLLOW while preserving handle-relative I/O semantics. Co-Authored-By: heihutu <[email protected]> * fix(ecstore): delete empty leased buckets via dirfd Use the held mount lease fd as the parent for non-force empty bucket deletion on Linux so procfd-rooted paths do not get rejected as BucketNotEmpty. Also make the download-part OpenOptions truncate behavior explicit and keep fsync test recording stable across procfd canonicalization. Co-Authored-By: heihutu <[email protected]> * fix(ecstore): scan leased bucket paths for emptiness Use the local disk I/O root for bucket emptiness probes before non-force bucket deletion and table-bucket metadata checks. This keeps validation on the same mount instance as the subsequent local disk delete path. Co-Authored-By: heihutu <[email protected]> * test(ecstore): align lease path test probes Co-Authored-By: heihutu <[email protected]> * fix(heal): block unsafe replacement recovery restarts Co-Authored-By: heihutu <[email protected]> * fix(heal): defer blocked replacement candidates Co-Authored-By: heihutu <[email protected]> * fix(heal): retry transient replacement discovery Co-Authored-By: heihutu <[email protected]> * fix(heal): keep transient recovery errors retryable Co-Authored-By: heihutu <[email protected]> * fix(heal): block corrupt legacy replacement state Co-Authored-By: heihutu <[email protected]> * fix(heal): classify flat replacement intent corruption Co-Authored-By: heihutu <[email protected]> * fix(heal): keep transient resume loads retryable Classify malformed legacy replacement state as blocking corruption while preserving disk and transient load failures for retry. This avoids permanently blocking replacement recovery on temporary storage errors. Co-Authored-By: heihutu <[email protected]> * fix(heal): avoid latching transient legacy publishes Co-Authored-By: heihutu <[email protected]> * fix(heal): retry blocked legacy migrations Co-Authored-By: heihutu <[email protected]> * fix(heal): defer blocked startup recoveries Co-Authored-By: heihutu <[email protected]> * fix(ecstore): preserve disk sync limiter across lease roots Co-Authored-By: heihutu <[email protected]> --------- Co-authored-by: heihutu <[email protected]> Co-authored-by: zhi22915 <[email protected]>
This commit is contained in:
houseme
co-authored by
heihutu
zhi22915
heihutu
zhi22915
parent
10a1d6b6e6
commit
f17ea7f146
@@ -213,6 +213,16 @@ missed work. Scanner-originated object checks should appear under
|
||||
under `admin/high`. If scanner work grows but admin work remains blocked, treat
|
||||
that as heal queue pressure rather than scanner pacing pressure.
|
||||
|
||||
## Replacement Recovery Completion
|
||||
|
||||
`POST /v3/background-heal/status` is an execution-queue view. `state=idle`, zero queue and active counts, an online disk, a readable object, or acceptance of an Admin deep-heal request do not independently prove that a replacement disk contains every erasure shard.
|
||||
|
||||
Treat replacement recovery as verified only after the repair task has completed for the exact replacement instance and an operator has confirmed the target disk contains the expected `xl.meta` and data parts for every relevant object version. A replacement that is not mounted, is unsafe to format, loses its marker, or returns a partial target outcome must be treated as deferred or incomplete rather than complete.
|
||||
|
||||
The v3 route and its peer status protocol preserve their existing fields for mixed-version clusters. A new node must not infer replacement completion from an old or unavailable peer; regard that information as unknown or degraded until every required peer can report the same replacement instance and verified completion. Do not automate destructive replacement actions from an `idle` observation alone.
|
||||
|
||||
Replacement resume and checkpoint files use an independent on-disk schema. A newer reader rejects a future schema rather than continuing with data it cannot interpret, while an older binary cannot safely enforce the new generation fence because it may ignore fields it does not know. Do not roll a cluster back after a replacement generation has started. Complete that recovery with the current-or-newer release; if it cannot complete, keep that version for diagnosis rather than deleting its durable records or continuing with an older binary.
|
||||
|
||||
## Reading Replication Repair
|
||||
|
||||
`metrics.replication_repair`, `metrics.current_cycle_replication_repair`, and
|
||||
|
||||
Reference in New Issue
Block a user