* fix(scanner): defer usage publication during pool recovery
* fix(scanner): preserve metrics when publication is deferred
* fix(scanner): route test types through storage boundary
* fix(scanner): keep cache floor deferred during movement
* fix(lifecycle): safely expire all object versions
* fix(lifecycle): preserve delete-all replication purges
* fix(lifecycle): remove dead replication journal
* fix(ci): avoid lifecycle transition test stack overflow
* fix(lifecycle): release recovery locks before tier IO
* test(lifecycle): align object-lock error assertions
* test(lifecycle): avoid scanner restore stack overflow
* test(scanner): avoid stack overflow in transition and restore flow test (#6300)
* refactor(scanner): split remote_scanner.rs into stream child module (#6289)
Split the 3080-line remote_scanner.rs (47% inline tests) into a
canonical foo.rs + foo/ module tree with zero behavior change:
- remote_scanner.rs (~320): protocol constants, process statics, and
the request decode/validate/admit/preflight/claim API plus root
re-exports
- remote_scanner/stream.rs (~1340): wire/frame types, replay cache,
FrameAuthenticator, serve path, local bucket scan + persist, client
scan, and the bounded stream plumbing
- remote_scanner/stream/tests.rs (~1470): the inline test module as a
child module of stream so it can reach both parents' private items
All crate paths are unchanged: lib.rs re-exports
(serve_remote_scanner_request, RemoteScannerRequest, ...) resolve
through root re-exports, and scanner_io's crate::remote_scanner::
{scan_remote_bucket, RemoteScannerScanSpec, RemoteScannerOutcome}
paths resolve through pub(crate) re-exports. Cross-module items gain
pub(super), whose scope equals the old single-module privacy domain;
no item's effective visibility widens. Code is moved verbatim apart
from those markers, per-module import headers, and rustfmt line
re-wraps.
Co-authored-by: heihutu <[email protected]>
* refactor(heal): split resume.rs into focused child modules (#6290)
Split the 4242-line resume.rs (46% inline tests) into a canonical
foo.rs + foo/ module tree with zero behavior change:
- resume.rs (~1020): state file constants, PersistThrottle, ResumeState,
ResumeManager core (constructors, load/discovery, progress mutators,
ordinary persistence) plus root re-exports
- resume/replacement.rs (~690): replacement-intent/proof types and the
ResumeManager replacement-lifecycle methods
- resume/checkpoint.rs (~350): ResumeCheckpoint + CheckpointManager
- resume/utils.rs (~310): ResumeUtils statics
- resume/tests.rs (~1980): the inline test module as a child module
All module paths are unchanged (heal::resume::CheckpointManager and
friends resolve through root re-exports), so no consumer inside or
outside the crate changes. Items defined in child modules keep
module-private visibility; only the ten cross-module helpers gain
pub(super), which is not part of the crate API. Code is moved verbatim
apart from those visibility markers, four super::storage_api path
fixes, and the new per-module import headers.
Co-authored-by: heihutu <[email protected]>
* refactor(scanner): split scanner_io.rs into child modules (#6294)
Split the 5369-line scanner_io.rs (39% inline tests) into a canonical
scanner_io.rs + scanner_io/ module tree with zero behavior change:
- scanner_io.rs (~660): constants, metadata-error constructors, the
bucket scan plan, cycle-status classification helpers, the ScannerIO /
ScannerIOCache / ScannerIODisk traits, and ScannerCycleResult
- scanner_io/dirty_usage.rs (~300): process-wide dirty-usage statics
and the acknowledgment protocol
- scanner_io/guards.rs (~270): concurrency gauges and RAII guards
- scanner_io/cache.rs (~410): scanner cache locks and the snapshot
persist/publish path
- scanner_io/io_cycle.rs (~390), io_cache.rs (~1160), io_disk.rs
(~230): the ECStore / SetDisks / Disk trait implementations
- scanner_io/publish_gate_tests.rs (~750) and tests.rs (~1340): the two
inline test modules as child modules
All crate paths are unchanged: the lib.rs scanner_io re-exports and
every crate::scanner_io:: consumer (scanner.rs, remote_scanner,
scanner_folder, and cross-crate rustfs users) resolve through root
re-exports with their original visibilities (pub stays pub, pub(crate)
stays pub(crate)). Cross-module items gain pub(super), whose scope
equals the old single-module privacy domain. Code is moved verbatim
apart from those markers, per-module import headers, and rustfmt
re-wraps.
The logging-guardrail nsscanner_disk skip-set_disks rule now points at
scanner_io/io_disk.rs where the function moved; the pattern and
thresholds are unchanged.
Co-authored-by: heihutu <[email protected]>
* refactor(scanner): split data_usage_define persistence and tests (#6292)
Split the 3655-line data_usage_define.rs (59% inline tests) into a
canonical foo.rs + foo/ module tree with zero behavior change:
- data_usage_define.rs (~950): cache constants and revision helpers,
the data-usage tree types, DataUsageCacheInfo with its hand-written
Serialize, the in-memory tree operations, dui, and marshal/unmarshal
- data_usage_define/persistence.rs (~580): the load/backup/restore
ladder (load, try_load_inner, revision_for_path) and the CAS save
path with its retry policy and save metrics
- data_usage_define/tests.rs (~2155): the inline test module as a child
module
All module paths are unchanged (the lib.rs data_usage_define::* glob
re-export and every crate::data_usage_define:: consumer resolve as
before). The hand-written map-encoded Serialize for
DataUsageCacheInfo is moved byte-for-byte per the AGENTS.md
cross-cutting invariant; on-disk names and the cache key format const
stay in the root. Four persistence helpers used by tests gain
pub(super), whose scope equals the old single-module privacy domain.
Code is moved verbatim apart from those markers, per-module import
headers, and rustfmt re-wraps.
Co-authored-by: heihutu <[email protected]>
* chore(deps): bump datafusion to 55.0.0 (#6288)
* refactor(heal): split task.rs per heal kind (#6293)
* feat(ecstore): batch small file fdatasync commits (#6297)
* feat(ecstore): batch small file fdatasync commits
Add a default-off experimental file fdatasync group commit path for small rename_data shard directories. The coordinator batches same-disk waiters into one blocking task while preserving per-directory source fsync after shard contents are durable.
Co-Authored-By: heihutu <[email protected]>
* test(e2e): wait for compression S3 readiness
Reuse the shared S3 API readiness probe for compression test servers so multipart requests do not race the startup readiness gate after the TCP port opens.
Co-Authored-By: heihutu <[email protected]>
---------
Co-authored-by: heihutu <[email protected]>
* fix(tier): recover multi-committed mutation intents (#6296)
* fix(tier): recover multi-committed mutation intents
* fix(tier): recover committed mutations on standalone nodes
* test(scanner): avoid stack overflow in transition test
---------
Co-authored-by: heihutu <[email protected]>
Co-authored-by: cxymds <[email protected]>
---------
Co-authored-by: houseme <[email protected]>
Co-authored-by: heihutu <[email protected]>
* fix(ecstore): single-flight remote disk recovery
* test(ecstore): cover remote recovery review cases
* test(ecstore): exercise recovery through disk slot
* test(ecstore): match format reads exactly
* test(ecstore): cover recovery teardown races
* style(ecstore): format recovery race tests
* fix(ecstore): remove unused health snapshot helper
* fix(ecstore): group recovery monitor test state
* fix(protos): preserve production source in compatibility checks
