Commit Graph
20 Commits
Author SHA1 Message Date
b36a730e48 feat(scanner): expose checkpoint and source work status (#3230)
* feat(scanner): expose checkpoint and source work status

* fix(scanner): count ignored checkpoints once per scan

---------

Co-authored-by: Henry Guo <[email protected]>
2026-06-06 10:47:39 +00:00
0d445afda9 feat(scanner): add versioned scan checkpoints (#3220)
Co-authored-by: Henry Guo <[email protected]>
2026-06-05 13:49:41 +00:00
542720a1f7 feat(scanner): add partial scan resume hints (#3207)
* feat(scanner): add partial scan resume hints

* test(scanner): cover clearing scan resume hints

* fix(scanner): apply resume hint to combined child order

---------

Co-authored-by: Henry Guo <[email protected]>
2026-06-04 09:21:30 +00:00
6850457707 feat(scanner): add runtime scanner controls and status (#3203)
* feat(scanner): add runtime scanner controls and status

* fix(scanner): validate persisted scanner config

* docs(scanner): clarify start delay behavior

---------

Co-authored-by: Henry Guo <[email protected]>
Co-authored-by: houseme <[email protected]>
2026-06-04 05:16:26 +00:00
cc07946782 feat(scanner): add cycle budget observability (#3166)
* feat(scanner): add cycle budget observability

* fix(scanner): clear cycle state after budget stop

* test(scanner): stabilize timeout-based scanner tests

* fix(scanner): keep cycle ILM counts scanner-only

* test(scanner): avoid test-only pending import

---------

Co-authored-by: Henry Guo <[email protected]>
Co-authored-by: houseme <[email protected]>
Co-authored-by: 安正超 <[email protected]>
2026-06-03 03:32:41 +00:00
a14657f517 fix(scanner,data-usage): fix add() logic inversion and usize underflow in reduce_children_of (#3142)
* fix(scanner,data-usage): fix add() logic inversion and usize underflow in reduce_children_of

Bug #1: add() had inverted logic — returned early when children were present,
making the recursive child traversal dead code. Leaf compaction path was
completely non-functional. Fixed by recursing into children when present and
collecting leaves only at leaf nodes.

Bug #2: reduce_children_of used  which underflows in debug
(panic) or release (wraps to usize::MAX, compacts entire cache). Fixed with
saturating_sub.

Both bugs existed identically in crates/scanner and crates/data-usage.

Added 5 tests covering add() subtree traversal, edge cases, and
reduce_children_of compaction behavior.

Closes rustfs/backlog#652

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix: collect internal nodes as compaction candidates, not leaves

The previous fix for add() collected leaf nodes (children empty) as
compaction candidates. This is incorrect: total_children_rec returns 0
for leaves, so compacting them never decrements  — the
compaction loop makes no progress.

Collect internal nodes (children non-empty) instead. Compacting an
internal node removes its subtree via delete_recursive, which actually
reduces the total child count. Leaf nodes are skipped since they have
no children to remove.

Updated tests to match the corrected semantics.

* refactor: rename leaves→candidates, add data-usage regression tests

- Rename misleading  variable/parameter to  in
  add() and reduce_children_of() for both crates
- Add 4 regression tests in crates/data-usage covering add() candidate
  selection and reduce_children_of() compaction + saturating_sub

* fix: address data usage compaction review feedback

---------

Co-authored-by: Claude Opus 4.8 <[email protected]>
Co-authored-by: houseme <[email protected]>
2026-05-31 21:52:53 +00:00
92104cb354 fix(scanner): reduce single-disk scanner churn (#3135)
* fix(scanner): slow single-disk default scans

* fix(scanner): reduce single-disk scanner churn

---------

Co-authored-by: Henry Guo <[email protected]>
Co-authored-by: houseme <[email protected]>
2026-05-31 05:05:04 +00:00
housemeandGitHub 73bde843d6 refactor(s3): consolidate semantic boundaries and remove s3-common (#3012)
* refactor(common): introduce rustfs-data-usage core crate

* refactor(concurrency): migrate workers crate into concurrency

* refactor(crypto): migrate appauth token APIs into crypto

* fix docs urls

* remove unused crate

* refactor(data-usage): switch consumers to rustfs-data-usage

* chore(fmt): apply cargo fmt and lockfile sync

* refactor(common): remove data_usage compatibility re-export

* refactor(capacity): move capacity_scope to object-capacity

* refactor(io-metrics): relocate internode metrics from common

* refactor(common): decouple scanner report from madmin

* chore(fmt): normalize import ordering after pre-commit

* refactor(s3): split s3 types and ops crates

* refactor(s3): centralize event version and safe parsing

* refactor(s3): add op-event compatibility guardrails

* refactor(s3): add runtime op-event mismatch observability

* refactor(s3): extract delete event mapping helper

* refactor(s3): extract put event mapping helper

* refactor(s3): consolidate remaining event semantic helpers

* refactor(s3): add op-event coverage checks and observability alerts

* refactor(s3-ops): consolidate op-event semantic mapping

* refactor(scanner): remove last_minute wrapper module

* refactor(scanner): consolidate duplicated data usage models
2026-05-19 12:50:25 +00:00
housemeandGitHub 9677320f23 fix(scanner): stabilize data usage cache persistence under slow metadata I/O (#2594) 2026-04-18 16:36:28 +00:00
housemeandGitHub 478720d2ee Centralize lifecycle state updates and fix systemd running status (#2567) 2026-04-17 03:20:11 +00:00
579b124726 lint: clippy rules or_fun_call (#2561)
Co-authored-by: houseme <[email protected]>
2026-04-16 02:48:39 +00:00
TungliesandGitHub 49366ee200 chore(lint): clippy rules redundant_clone (#2554) 2026-04-15 13:54:07 +00:00
weisdandGitHub b9b7d86ae4 feat: improve legacy metadata and admin compatibility (#2202) 2026-03-18 21:05:09 +08:00
weisdandGitHub ed18b3da75 Fix data usage cache and scanner (#2074) 2026-03-04 19:55:01 +08:00
e73b17aff6 fix(scanner): skip recent IO-error objects (#1860)
Signed-off-by: LoganZ2 <[email protected]>
Co-authored-by: houseme <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: heihutu <[email protected]>
Co-authored-by: loverustfs <[email protected]>
Co-authored-by: 安正超 <[email protected]>
2026-02-27 22:25:52 +08:00
682b5bbb2f perf(scanner): Change DataUseageEntry from clone to borrow (#1757)
Signed-off-by: evan slack <[email protected]>
Signed-off-by: houseme <[email protected]>
Co-authored-by: loverustfs <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: houseme <[email protected]>
2026-02-10 08:53:26 +08:00
dce117840c refactor: NamespaceLock (nslock), AHM→Heal Crate, and Lock/Clippy Fixes (#1664)
Co-authored-by: Copilot <[email protected]>
Co-authored-by: weisd <[email protected]>
Co-authored-by: houseme <[email protected]>
2026-01-30 13:13:41 +08:00
d251b9fb35 fix: unify path handling to use S3-standard forward slashes on all platforms (#1555)
Signed-off-by: houseme <[email protected]>
Signed-off-by: 安正超 <[email protected]>
Co-authored-by: heihutu <[email protected]>
Co-authored-by: LeonWang0735 <[email protected]>
Co-authored-by: loverustfs <[email protected]>
Co-authored-by: 安正超 <[email protected]>
Co-authored-by: Copilot <[email protected]>
2026-01-26 18:49:21 +08:00
housemeandGitHub e5284a85ed fix: Fixed detection warnings in rust v1.93.0 (#1591) 2026-01-23 13:09:41 +08:00
weisdandGitHub 6631407416 feat: Add RustFS Scanner Module and Multiple Bug Fixes (#1579) 2026-01-22 13:39:38 +08:00