feat(scanner): expose prefix-level bucket usage via admin API
The scanner's per-bucket, per-set usage caches already hold a path-keyed
prefix tree, but dui() flattened it only to bucket names — consoles and
operators had no way to ask "what does this prefix hold" without an S3
listing sweep (rustfs/backlog#1872, MinIO loadPrefixUsageFromBackend
parity).
Add:
- data-usage: prefix_usage_in_cache — a shared aggregation over the
entry map (arbitrary prefix, full counters, one-level sub-prefix
breakdown with names recovered from the literal-path cache keys),
hardened like the scanner's checked flatten: cycles, dangling child
links, over-deep trees, and overflowing counters yield None rather
than unbounded recursion or wrapped totals.
- ecstore: ECStore::all_set_disks — iterate every erasure set so a
query can read each set's own cache copy; the hash-routed store path
would always land on one set.
- scanner: bucket_prefix_usage — per-set loads (5s budget each, a slow
set degrades to not-reporting instead of stalling the caller),
merged across sets with partial/compacted/truncated flags, served
from a bounded 30s cache (128 entries, hard-capped) that bucket
writes invalidate through the dirty-usage hook.
- admin: GET /rustfs/admin/v3/usage/{bucket}?prefix=&max-entries=
behind the same any-of gate as datausageinfo (DataUsageInfoAdminAction
OR ListBucketAction), rejecting unknown query parameters and
clamping max-entries to 1..=10000. Route registered in the policy
table (deferred MultipleActions, matching datausageinfo) and the
route matrix test.
Closesrustfs/backlog#1872.
Co-authored-by: heihutu <[email protected]>
Delete the dead ecstore save_data_usage_cache and the thin
DataUsageCache::marshal_msg it was the only caller of, drop the
Serialize derive (and the dead DataUsageCacheStorage trait with its
save path) from the thin projection types so no write path can exist
outside the scanner's canonical map-encoded writer, and pin the
persisted .usage-cache.bin wire bytes with cross-crate fixture tests
on both the scanner writer and the thin reader.
Refs rustfs/backlog#1828 (T1-T3).
SizeSummary::tier_stats was populated for every scanned object but
apply_scanner_size_summary dropped it, so per-tier usage never reached
DataUsageInfo. Wire it through the same merge chain repl_target_stats
already uses, up to DataUsageInfo::tier_stats.
DataUsageEntry used the derived MessagePack encoding, which serialises
structs as arrays: appending a field turns the whole cache into a decode
error for older readers, so mixed-version nodes would invalidate each
other's cache every scan cycle. Give it the same hand-written
map-encoded Serialize DataUsageCacheInfo already carries, and record the
invariant in AGENTS.md.
Widen TierStats counters from i32 to u64 so a tier past 2^31 versions
cannot make checked_merge reject an entire usage snapshot, and drop the
duplicate TierStats/AllTierStats definitions in the scanner crate in
favour of the data-usage ones.
* feat(scanner): expose checkpoint and source work status
* fix(scanner): count ignored checkpoints once per scan
---------
Co-authored-by: Henry Guo <[email protected]>
* fix(scanner,data-usage): fix add() logic inversion and usize underflow in reduce_children_of
Bug #1: add() had inverted logic — returned early when children were present,
making the recursive child traversal dead code. Leaf compaction path was
completely non-functional. Fixed by recursing into children when present and
collecting leaves only at leaf nodes.
Bug #2: reduce_children_of used which underflows in debug
(panic) or release (wraps to usize::MAX, compacts entire cache). Fixed with
saturating_sub.
Both bugs existed identically in crates/scanner and crates/data-usage.
Added 5 tests covering add() subtree traversal, edge cases, and
reduce_children_of compaction behavior.
Closesrustfs/backlog#652
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* fix: collect internal nodes as compaction candidates, not leaves
The previous fix for add() collected leaf nodes (children empty) as
compaction candidates. This is incorrect: total_children_rec returns 0
for leaves, so compacting them never decrements — the
compaction loop makes no progress.
Collect internal nodes (children non-empty) instead. Compacting an
internal node removes its subtree via delete_recursive, which actually
reduces the total child count. Leaf nodes are skipped since they have
no children to remove.
Updated tests to match the corrected semantics.
* refactor: rename leaves→candidates, add data-usage regression tests
- Rename misleading variable/parameter to in
add() and reduce_children_of() for both crates
- Add 4 regression tests in crates/data-usage covering add() candidate
selection and reduce_children_of() compaction + saturating_sub
* fix: address data usage compaction review feedback
---------
Co-authored-by: Claude Opus 4.8 <[email protected]>
Co-authored-by: houseme <[email protected]>
