* feat(object-data-cache): close write/delete-side invalidation gaps The object data cache exposed only a single per-(bucket,object) invalidation primitive and no write-side ecstore hook, so several delete paths left dead bodies resident until TTL (hygiene/capacity, not stale-serving: lookups follow a fresh metadata quorum and cannot serve a gone object). This adds the missing primitives and wires them in. ODC-26 (backlog#1131): add an `ObjectMutationHook` trait beside the GET body hook, registered next to it at startup, and call it from the ecstore-internal delete paths (`apply_expiry_on_non_transitioned_objects`, `expire_transitioned_object` including the restored-copy branch, and `delete_object_versions`). The app impl is one `invalidate_object` call under a new `AfterLifecycleExpiry` reason. ODC-27 (backlog#1132): force prefix delete now invalidates the whole prefix, not just the prefix string. `store.delete_object(delete_prefix)` returns no deleted-name list, so this uses a new prefix primitive rather than the batch path. ODC-28 (backlog#1133): DeleteBucket now flushes the bucket via a new bucket-scope primitive (covers force and non-force, which share the delete_bucket call). ODC-C2 (backlog#1143): add `ObjectDataCache::clear()` and two admin handlers (GET stats, POST flush) routed through admin runtime_sources. The starshard identity index gains a single `remove_matching` full-scan API backing prefix/bucket/clear; it is documented as admin/delete-path only and never runs on the GET or fill hot path. New invalidation reasons and metric labels added; outcome (removed/noop) labelling kept correct for every new primitive. Also fixes a pre-existing broken intra-doc link in memory.rs. Co-Authored-By: heihutu <[email protected]> * refactor(ecstore): extract the shared HookSlot behind both cache hooks This PR introduced object_mutation_hook.rs by mirroring body_cache_hook.rs, which left two process-global registration slots whose register/get/clear bodies were line-for-line identical except the trait type and the WARN string: a RwLock<Option<Arc<dyn _>>>, an Arc::ptr_eq "different instance" warning, the poison-recovery closure, and the same read-lock-and-clone read. Two copies of the same swap-vs-warn logic can drift apart under maintenance. Hoist it into a generic HookSlot<T: ?Sized> that owns the logic once. Each hook module keeps its `static HOOK: HookSlot<dyn XxxHook>` and its thin, unchanged public wrappers (register_/get_/clear_), so the crate's public surface and every call site are untouched — this is an internal consolidation, not a contract change. The load-bearing #1126 guarantee (newest registration wins, so a rebuilt AppContext is never stranded on a first-wins slot) previously had no direct test — the hook tests only covered register-then-notify. HookSlot now has its own unit tests including re_registration_swaps_to_the_latest_instance; mutation-testing confirms a first-wins regression fails exactly that test. No behavior change: the two hooks' existing tests, the P0 body_cache_hook_e2e regressions, and the app-layer mutation-hook tests all pass unchanged. Refs: backlog#1126, backlog#1131 Co-Authored-By: heihutu <[email protected]> * fix(admin): register the object-data-cache routes in the policy inventory This PR added GET /object-data-cache/stats and POST /object-data-cache/flush but did not list them in the two registries that must account for every admin route: the route-policy inventory (route_policy.rs) and the route matrix (route_registration_test.rs). Their coverage tests — route_policy_inventory_covers_registered_routes and test_admin_route_matrix_matches_registered_routes — failed on CI because a registered route had no policy/matrix entry. These two tests are not part of `make pre-commit` (which runs fmt + arch + quick-check, not the full suite), so the gap passed local pre-commit and only surfaced in the CI Test-and-Lint lane. stats is a read (ServerInfoAdminAction, Sensitive); flush mutates (ConfigUpdateAdminAction, High) — matching the actions the handlers already enforce. The MinIO-alias matrix test is unaffected: these are native rustfs endpoints with no MinIO equivalent. Refs: backlog#1143 Co-Authored-By: heihutu <[email protected]> --------- Co-authored-by: heihutu <[email protected]>
105 lines
3.8 KiB
Rust
105 lines
3.8 KiB
Rust
// Copyright 2024 RustFS Team
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
//! ecstore-facing write-side cache hook.
|
|
//!
|
|
//! Registered into ecstore so its internal delete paths (lifecycle/scanner
|
|
//! expiry, noncurrent-version cleanup, restored-copy expiry) can drop the
|
|
//! object's cached body the moment the object is removed, instead of leaving
|
|
//! dead bytes resident until TTL (ODC-26, backlog#1131).
|
|
|
|
use crate::app::object_data_cache::ObjectDataCacheAdapter;
|
|
use crate::storage::storage_api::ecstore_object::{ObjectMutationHook, register_object_mutation_hook};
|
|
use rustfs_object_data_cache::{ObjectDataCacheIdentity, ObjectDataCacheInvalidationReason};
|
|
use std::sync::Arc;
|
|
|
|
/// Adapter-backed implementation of ecstore's object mutation hook.
|
|
pub(crate) struct ObjectDataCacheMutationHook {
|
|
adapter: Arc<ObjectDataCacheAdapter>,
|
|
}
|
|
|
|
/// Registers the mutation hook into ecstore. No-op for a disabled cache so the
|
|
/// delete paths keep a single `None` branch when the feature is off, matching
|
|
/// [`register_object_data_cache_body_hook`](super::register_object_data_cache_body_hook).
|
|
pub(crate) fn register_object_data_cache_mutation_hook(adapter: Arc<ObjectDataCacheAdapter>) {
|
|
if adapter.is_disabled() {
|
|
return;
|
|
}
|
|
register_object_mutation_hook(Arc::new(ObjectDataCacheMutationHook { adapter }));
|
|
}
|
|
|
|
#[async_trait::async_trait]
|
|
impl ObjectMutationHook for ObjectDataCacheMutationHook {
|
|
async fn after_object_mutation(&self, bucket: &str, object: &str) {
|
|
let _ = self
|
|
.adapter
|
|
.invalidate_object(
|
|
ObjectDataCacheIdentity::new(bucket, object),
|
|
ObjectDataCacheInvalidationReason::AfterLifecycleExpiry,
|
|
)
|
|
.await;
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
use bytes::Bytes;
|
|
use rustfs_object_data_cache::{
|
|
ObjectDataCacheBodyVariant, ObjectDataCacheConfig, ObjectDataCacheFillResult, ObjectDataCacheGetRequest,
|
|
ObjectDataCacheLookup, ObjectDataCacheMode,
|
|
};
|
|
|
|
fn fill_enabled_adapter() -> Arc<ObjectDataCacheAdapter> {
|
|
Arc::new(
|
|
ObjectDataCacheAdapter::new(ObjectDataCacheConfig {
|
|
mode: ObjectDataCacheMode::FillMaterializeEnabled,
|
|
max_bytes: 8_388_608,
|
|
// Fill must not depend on the live memory reading (host vs container).
|
|
min_free_memory_percent: 0,
|
|
..ObjectDataCacheConfig::default()
|
|
})
|
|
.expect("adapter"),
|
|
)
|
|
}
|
|
|
|
fn request<'a>(bucket: &'a str, object: &'a str) -> ObjectDataCacheGetRequest<'a> {
|
|
ObjectDataCacheGetRequest {
|
|
bucket,
|
|
object,
|
|
version_id: None,
|
|
etag: "etag",
|
|
size: 5,
|
|
body_variant: ObjectDataCacheBodyVariant::FullObjectPlainV1,
|
|
}
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn mutation_hook_invalidates_cached_body() {
|
|
let adapter = fill_enabled_adapter();
|
|
let plan = adapter.plan_get(request("b", "k"));
|
|
assert_eq!(
|
|
adapter.fill_body(&plan, Bytes::from_static(b"hello")).await,
|
|
ObjectDataCacheFillResult::Inserted
|
|
);
|
|
|
|
let hook = ObjectDataCacheMutationHook {
|
|
adapter: Arc::clone(&adapter),
|
|
};
|
|
hook.after_object_mutation("b", "k").await;
|
|
|
|
assert!(matches!(adapter.lookup_body(&plan).await, ObjectDataCacheLookup::Miss));
|
|
}
|
|
}
|