The Local backend's durability argument rests on properties of the filesystem it runs on, and those properties were assumptions: the crate had no test touching symlinks, none asserting a published file's mode, and none on directory replacement or cross-device behavior. Writing that verification surfaced three gaps.
The key directory's own permissions were never set and never read. create_dir_all applies the process umask, which is 0 in a good many container images, and the platform picks the mode far more often than an operator does: kubelet creates an emptyDir 0777, several PVC provisioners mkdir -m 0777, a --tmpfs mount lands at 1777. Write access there is the power to delete a key, destroying every object it protects, or to plant a record for a key id that does not exist yet. The directory is now created 0700 through DirBuilder::mode, so intermediate components are covered and no create-then-chmod window exists, and anything wider is narrowed on every start and re-read to confirm it took. Narrowing rather than refusing matches what the observability stack already does with its own directory; refusing would turn each of those platform defaults into a server that will not start while leaving the exposure on disk. Only a directory this process cannot secure is fatal.
An unspecified file_permissions meant whatever the umask said. The field is optional in the persisted configuration and stays optional, but with it absent the entire mode-application block was skipped, so under a 0 umask master key records were published world-readable. Absent now resolves to owner-only inside the commit protocol rather than at each call site, so the backup restore path — which passed the unset value straight through and published a legacy cluster's restored records at 0644 — is covered by construction.
Startup left symlinked commit temps behind forever, because the orphan sweep required a regular file. The protocol only ever creates temps with create_new, so an entry wearing a temp name and any other file type is either its own leftover or something planted.
Eight tests pin the boundaries: that a requested mode survives the umask, that an absent one still resolves owner-only, that a directory at each mode a real platform produces is narrowed, that publishing replaces a symlink instead of writing through it on both the hard_link and rename paths, that a planted hard link cannot be adopted as a key record, that a symlinked commit temp is removed without harming the key it pointed at, and that commit temps never leave the destination's directory. A real cross-device operation and a directory swapped between rename and fsync cannot be verified without a second filesystem and directory file descriptors respectively; both are recorded in the operations documentation rather than left looking covered.
Adding a KMS route already fails `route_registration_test`, and `route_policy` pins each route's action and risk with individual assertions, but nothing records the surface as a whole. A change to an existing route's action or risk therefore lands as an edited assertion rather than as a visible before/after — and these are the routes where that matters, since the action decides who may reach key material and the risk level decides which confirmations the route demands.
Every field is derived from ADMIN_ROUTE_POLICY_SPECS, so the snapshot cannot drift from the routing table; there is nothing to keep in sync by hand. A second test asserts the snapshot actually covers the surface it claims to: every listed route must be under /kms/ and must gate on a dedicated kms:* action, so a KMS route registered outside the policy table or falling back to a generic admin action cannot leave the snapshot green.
Per-key authorization scoping is deliberately not restated here. It is enforced and tested where it is implemented, by single_key_endpoints_reject_a_key_outside_the_policy_scope in handlers::kms_keys; a second hand-maintained list would be a claim nothing checks.
A key that cannot be described was handled two incompatible ways. Vault KV2 swallowed every describe failure and dropped the key from the page, so a damaged or newer-format record silently disappeared from the operator's inventory and from the deletion sweep's census. Local failed the whole listing instead, so one bad record stopped every scheduled deletion on the node for as long as the damage lasted. Both force a per-key problem into a whole-page answer.
ListKeysResponse now carries unreadable_key_ids, and the backends that read local key records classify per-key failures in one place: KeyNotFound is a concurrent deletion and is skipped, a material-level error names the key on the page, and anything else fails the listing, because it says nothing about a particular key and reporting it as key damage would turn a backend outage into a false data-loss alarm. A listing that covered the entire key set and found nothing readable still fails, since an empty page there is indistinguishable from a deployment with no keys; the guard is scoped to a page with no successor so a damaged key can never strand the keys behind it. The deletion sweep destroys the expired keys it can read, counts the unreadable ones, and withholds its lifecycle gauges rather than publishing a census over a key set it did not fully see.
Vault Transit needs the same treatment and is easy to miss: its per-key metadata records live in KV2 too, so folding every non-404 failure into a backend error left its per-key classification unreachable and one metadata record written by a newer build still failed every listing on the node.
Vault KV2 record reads gain the typed errors this needs: an unparseable body is MaterialCorrupt and an absent data envelope is MaterialMissing, where both were previously indistinguishable from Vault being unreachable. Only the parse failure's category and position are reported, because serde's own message embeds the offending scalar and that message reaches a log line and an admin HTTP body.
The admin list handlers refuse a malformed limit with 400 instead of silently substituting the default page size, and every page is capped at 1000 where it is cut, so a single request can no longer fan out one metadata lookup per key without bound. The four operation-level KMS metrics gain a backend label, since operation names are shared across backends and a Transit latency regression was previously indistinguishable from an AWS one. The Static backend captures its reported creation date once instead of reading the clock on every describe and list. POST /kms/clear-cache gains a named response type with an unchanged wire shape.
The namespace REST contracts PR (#5745) added 7 new s3_error! invocations in
read_bounded_json_body, pushing the s3s footprint counter from 1687 to 1693
and violating the ratchet baseline.
Replace those calls with ApiError::invalid_request() (gateway-side error
abstraction, rustfs/backlog#1677 F1, rustfs/backlog#1733) and lower the
baseline from 1687 to 1686.
- Add ApiError::invalid_request(message) constructor to rustfs/src/error.rs
- Replace 7 s3_error! calls in read_bounded_json_body with S3Error::from(ApiError)
- Lower S3_ERROR_LINES_BASELINE from 1687 to 1686
Verification:
- make pre-commit: all guard scripts + fmt-check + quick-check passed
- make clippy-check: passed
- cargo test table_catalog + admin handler tests: 406/406 passed
- s3s-e2e: 27/27 passed
Freeze the direct s3s dependency surface with a lower-only ratchet so it
cannot grow while the s3gate/gateway migration shrinks it
(rustfs/backlog#1677 review finding F1; acceptance criteria in
rustfs/backlog#1733). Baselines verified on 2026-08-05: 236 files
importing s3s, 1686 s3_error! invocation lines. Wired into make
pre-commit / pre-pr / dev-check and the Quick Checks job in ci.yml and
its ci-docs-only.yml mirror.
PR #5719 fixed the issue #5716 per-object heal log amplification (per-object statements demoted, heal spans forced to TRACE, raw metadata dumps banned by guardrail) and superseded the demotion originally proposed here. This PR now carries only the residual cleanup on top of it:
- Convert the remaining bare-field and format-arg heal logs in crates/ecstore/src/set_disk/ops/heal.rs to the file's structured convention (event/component/subsystem + context fields): missing-object skip, disk-marked-for-healing, cannot-reconstruct errors, dangling-cleanup error, missing data_dir error, xl.meta regeneration warn, and orphan-reclaim failure warn.
- Demote the last remaining info! in the file — the per-set heal_format "set disk formats success, NoHealRequired" no-op message — to a structured debug! (error_count instead of a raw errs dump), and drop its whitelist exclusion in scripts/check_logging_guardrails.sh so the no-INFO check for set-disk heal files is strict.
No control flow or behavior changes.
Multipart upload ids embed the process-global deployment id at both create time and list time. Under plain cargo test (thread-parallel, shared process globals) a concurrently running test that re-initializes a store can swap the global between the two reads, making full-upload-id equality assertions fail spuriously (observed: core::sets::tests::list_multipart_uploads_merges_all_sets_without_pagination_loss failing when run concurrently with bucket::quota tests, passing in isolation).
Add a test-only upload_uuid_suffix helper next to deployment_upload_id and make the affected assertions compare only the decoded <uuid>x<timestamp> suffix. Where suffix normalization changes within-key ordering (base64 alphabet order is not byte order), both sides are sorted before comparison. nextest/CI is unaffected (process-per-test); this only hardens local plain cargo test runs.
initialize_local_disk_maps appended pool entries to local_disk_set_drives and inserted into local_disk_map without ever clearing previous state. Every caller (both production startup entry points and all tests) passes the FULL topology, so re-initializing the same InstanceContext left the pool/set vectors sized for the stale topology and panicked with index-out-of-bounds for wider disk indices.
This surfaced as deterministic cross-test contamination under single-process cargo test: in crates/heal heal_b920_subquorum_union_test, a 4-disk test initialized pool 0 as [None; 4] on the process-level default context, and the two 8-disk tests then panicked at disk_idx 4. Each test passed alone, and CI never caught it because cargo nextest isolates every test in its own process.
Fix: clear both registries at the start of initialize_local_disk_maps so initialization is idempotent and last-topology-wins. Add a regression unit test in ecstore (process-isolation-proof, unlike the heal integration binary) that re-initializes the same context with a wider topology; it fails with the pre-fix code.
The multipart staging namespace is one flat set of sha256(bucket/object) directories shared by every bucket, and the cross-set listing rewrite reads every upload's metadata. Two shapes poisoned the whole ListMultipartUploads response with InternalError: Corrupted format: a healthy in-flight upload belonging to another bucket (its stored owner bucket fails the guard and fell into the corrupted-format arm), and a single upload directory whose xl.meta was torn by an unclean shutdown. Docker Distribution calls ListMultipartUploads on every PATCH/commit, so either shape broke OCI registry pushes entirely (issue #5716).
Foreign-bucket uploads are now skipped silently, and directories whose metadata is affirmatively corrupt at quorum are skipped with a debug log, while every other decode failure (quorum loss from offline disks, timeouts, transport errors) keeps failing the listing so clients retry instead of silently losing entries. The degrade-vs-propagate decision is a named corrupt-family classifier with a unit test pinning both sides. FileMeta::check_xl2_v1 now classifies a missing or wrong XL2 magic as FileCorrupt instead of an anonymous io error so damage is distinguishable from transient IO faults.
Refs #5716
#5724 reclaimed the synthetic inline-rollback dir after a committed rename with delete_data_dir(recursive: true), which has no notion of object metadata: for unversioned objects the synthetic UUID is a fixed, publicly-known constant, so object/<rollback-dir> can simultaneously be a legitimate child key's directory, and recursively deleting it reopens the authorization bypass #5703 closed (PutObject on K destroying K/<uuid> without DeleteObject permission).
Replace the recursive pass with a file-precise one: after quorum commit, delete exactly object/<rollback>/xl.meta.bkp with a non-recursive delete on every disk whose rollback dir is not also the cleanup dir. The parent-rmdir walk removes the dir only when the backup was its sole content, so the BucketNotEmpty leak fix is preserved (#5724's regression test passes unchanged) while a child key at the same path keeps its metadata. The undo path's restore_metadata_backup now also reclaims the emptied synthetic dir, mirroring restore_delete_rollback.
The fallback future embeds the whole snapshot loader, and every object write nests a quota check several futures deep, so inlining it grew each write's state machine by the loader's full size — the debug-build 2MiB worker-stack overflow class fixed for bucket-config writes in #5648. Box the fallback at its call site; the allocation only happens on the degraded path.
fix(heal): demote per-object logs and cap erasure-set failure warns
Follow-up to rustfs/rustfs#5716. Per-object heal task kinds (Object/Metadata/MRF/ECDecode) queued by MRF/autoheal/scanner loops emitted info!/warn!/error! lines per object: task lifecycle (started/completed/timed_out/failed), the missing-object warn, queue admission full/drop/displacement warns, retry-admission decisions, and uncapped per-object warns in erasure-set sweeps.
Add a shared demote_to_debug_when! macro that keeps aggregate task kinds and admin/internal requests at operator-visible levels while demoting per-object occurrences to debug!, sample-cap the erasure-set transient_skip/failed warns per bucket via take_failure_log_sample (reusing the heal_bucket_objects precedent), demote the per-retry admission decision logs to debug! (covered by rustfs_heal_admission_total and the scheduler task_retrying/task_failed events), and record the previously unmetered duplicate-admission outcome.
Extend scripts/check_logging_guardrails.sh with injection-verified regression guards and run it in both quick-checks jobs (ci.yml and its ci-docs-only.yml mirror).
A failed target construction previously logged only reason=construction_failed and pushed an opaque "target construction failed" summary, discarding the underlying TargetError. Debugging rustfs#5115 showed the log repeated every 5s with no root cause, even though the error carried an actionable egress-policy rejection (RUSTFS_OUTBOUND_ALLOW_ORIGINS hint).
The Err branch now includes the error detail in both the error! log (detail field) and the returned failures summary. The detail is scrubbed against the instance's merged config via the loader's existing per-field redaction (secrets -> ***redacted***, endpoint URLs -> origin only, DSNs -> password masked) so credential-bearing values never reach logs or Admin-visible summaries.
FileInfo's derived Debug printed the full metadata map (including X-Rustfs/X-Minio-Internal-Server-Side-Encryption-Sealed-Key and -Iv values, i.e. KEK-wrapped DEK ciphertext) and the full inline data bytes (plaintext user content for non-SSE small objects), so any whole-struct log dump such as the heal_object dumps leaked user data and sealed key material into logs.
Replace the derive with a manual Debug impl that redacts encryption metadata values (keys stay visible, values print as redacted with length) under both internal prefixes, and elides data/checksum bytes to a length summary. The exhaustive destructuring forces every future field through an explicit show/redact decision. starts_with_ignore_ascii_case is made pub in rustfs-utils for reuse.
#5703 split rollback state from old-data-dir cleanup so the synthetic inline-rollback dir is reported only as rollback_data_dir and never reclaimed as if it were a real data dir. But nothing reclaims it after a successful commit either: every overwrite of an inline version by a non-inline one leaves <object>/<rollback-dir>/xl.meta.bkp behind. The residue is not referenced by any version, so it survives object deletion and DeleteBucket fails with BucketNotEmpty forever — the mass teardown cascade currently failing the S3 Implemented Tests CI lane.
Reclaim the synthetic dirs in SetDisks::rename_data once the commit holds write quorum. The quorum-failure undo inside the same function is the only consumer of the backup, so its window is closed at that point. Best-effort with the same anti-misdelete posture as commit_rename_data_dir: never touch the just-committed data dir, and residue must not fail a durable write (backlog#898).
Upgrading from a pre-v2 release leaves only the legacy .usage.json snapshot, which has no completeness marker and is demoted to non-authoritative, so every write to a quota-enabled bucket failed closed with a retryable 503 until the scanner's first complete cycle persisted .usage.v2.json — a production outage on large namespaces (issue #5716).
Quota admission now degrades to the last persisted per-bucket size: normalize_loaded_data_usage returns the pre-discard bucket sizes, the TTL-bounded snapshot cache retains them (carried forward through failed refreshes), and QuotaChecker::get_real_time_usage falls back to that baseline when the authoritative caches miss. The baseline is static between snapshot loads, so hard-quota enforcement is advisory for the duration of the degraded window — strictly tighter than beta.11 (usage treated as 0) and strictly more available than a blanket 503. Buckets absent from every persisted snapshot still fail closed, and removing a bucket's usage from the backend purges the baseline so a recreated bucket cannot inherit the dead incarnation's size.
Refs #5716
Both per-target replication methods assign the optimistic Completed status to rinfo before building put options, and the Err branch of replication_put_object_options returned rinfo unchanged. Since #5633 made the source encryption classification case-insensitive, managed SSE sources are rejected at this gate, and the rejection was reported as successful replication: the source object was marked COMPLETED, ObjectReplicationComplete was emitted, and nothing existed on the target.
Set replication_status = Failed (and record the error in the replicate_object branch) so the composite status, the OperationFailedReplication event, and MRF retries reflect the fail-closed outcome. This restores the contract pinned by test_bucket_replication_sse_kms_failure_contract, which timed out in the e2e-replication-nightly runs on 2026-08-03 and 2026-08-04.
When a returning node carries a stale object version that was deleted on
the quorum, the heal disk-walk partial callback used `resolve_union`
which picks only one entry from divergent disk entries. The minority
version was never enumerated and therefore never cleaned up.
Replace `resolve_union` + `ingest` with a new `ingest_merged` that
collects all unique versions from every partial entry across disks,
deduplicating by (name, version_id). This ensures stale data on a
returning node is surfaced for healing and can be deleted as dangling.
Fixes#5029
When a fresh multi-node cluster starts, the first disk detects all disks
as unformatted and initializes the format. However, `should_init_erasure_disks`
and `quorum_unformatted_disks` only counted `UnformattedDisk` errors. Remote
peers that have not yet started their gRPC server return transient network
errors (connection refused, timeout) instead of `UnformattedDisk`, causing
the first disk to miss the "all unformatted" signal and creating a deadlock:
first disk retries endlessly while non-first disks wait for it.
Add `is_unformatted_or_transient_network` that treats transient network
errors as equivalent to `UnformattedDisk` for the bootstrap decision.
A remote disk that cannot be reached during fresh-cluster startup is
indistinguishable from an unformatted disk — the peer may simply not
have started its gRPC server yet.
Fixes#5655
Large buckets with millions of objects can take longer than the default
5-second walkdir timeout to produce the first page of listing results.
This causes timeouts in the web UI and mc CLI when opening or scanning
such buckets.
Skip the walkdir total timeout for S3 ListObjects operations when no
explicit walkdir_timeout is configured. The stall timeout (5s with no
forward progress) still protects against drives that stop responding.
This matches the scanner's existing behavior for the same reason.
Fixes#5647
Audit initialization requires the AppContext (server config + object
store) which is published by ensure_startup_after_iam inside
init_iam_runtime. Moving init_audit_runtime after init_iam_runtime
ensures the runtime sources are available when audit starts.
Fixes#5681
Co-authored-by: RustFS <[email protected]>
fix(ecstore): don't exclude pool when has_space_for is indeterminate
When `has_space_for` returns an error (not enough online disks to
reliably determine space), the old code treated this the same as
"definitely no space" via `unwrap_or_default()`, zeroing out the
pool's available capacity. During pool decommission this creates a
false "Disk full" (500) for S3 PUT requests:
- Pool 0 (decommissioning) is correctly suspended → available = 0
- Pool 1 (active) has some disks whose disk_info call fails under
heavy migration I/O → has_space_for returns Err → available forced
to 0
- get_available_pool_idx sees total = 0 → returns None → DiskFull
Fix: distinguish Ok(false) (genuinely full) from Err (indeterminate).
On Err, log a warning and fall through to compute available space from
whatever disks did respond. The actual write will enforce its own
quorum; a premature zero at the pool-selection layer is a false
rejection.
Closes#5497
* fix: address rc.1 release blockers
* fix: route release guards through architecture boundaries
* fix: close remaining rc.1 regression gaps
* refactor: group multipart listing options
* fix: resolve rc.1 CI regressions
* fix(ecstore): keep bucket-config writes off the caller's stack
A bucket-config write nests incarnation resolution (which can drive legacy
migration and a peer fan-out), a full metadata load, and `save` — itself an
object PUT that pulls in the whole erasure write path. Every request that
mutates bucket config is already several futures deep, so inlining all of
that into one state machine overflows the 2MiB worker stack in debug builds.
Two CI lanes aborted with SIGABRT on this:
ILM Integration (serial)
rustfs app::lifecycle_transition_api_test::
compensation_driven_complete_multipart_upload_still_transitions
Test and Lint (swift)
rustfs-protocols::swift_metadata_persistence::
swift_metadata_writes_are_durable
Neither test file is touched by this branch and both lanes are green on
main. Stack-pointer probing showed ~780KiB consumed between
`metadata_sys::update` and the config read alone, with single hops of
363KiB (`update` -> `acquire_config_write_guard_for_incarnation`), 125KiB
and 105KiB.
Box the deep sub-futures on both read-modify-write paths (`update` /
`update_checked` and `update_config_with` / `update_config_with_checked`)
so each guard's own state machine stays small. Behaviour is unchanged;
`update` -> guard drops to 253KiB and both tests pass on the default stack.
* fix(lifecycle): unbreak restore under the bucket generation fence
The ILM lane aborted on a stack overflow before reaching these, so they
were never reported; with that fixed, four restore tests fail. All four
are green on main and none of their test files are touched by this branch.
1. RestoreObject and ListMultipartUploads hard-required
`opts.expected_bucket_incarnation_id`, but `apply_bucket_generation_guard`
deliberately leaves it unset when no guard extension is present — only the
S3 access layer installs one. Every direct caller therefore got
`InternalError: ... bucket generation guard is missing`. Resolve the
current generation instead, the way the copy path already does. The fence
is unaffected: RestoreObject still re-reads the incarnation from disk and
compares before admitting the restore, and the multipart listing is
filtered by the value it resolves.
2. `restore_expiry_snapshot_matches` (new on this branch) rejected every
restored-copy expiry whose `restore_expires` had not already elapsed.
Whether the restored copy is due to expire is the ILM evaluator's
decision, made when it emitted DeleteRestoredAction; re-deriving it in
the set layer only adds a way for a legitimate action to be rejected.
The stale-event risk it appears to guard is already covered by the
surrounding snapshot match — a re-restore rewrites `restore_expires`,
so a replayed event fails the equality check. Drop the clause; the
fifteen identity clauses are unchanged.
Fixed:
rustfs app::lifecycle_transition_api_test::
restore_object_usecase_accepts_exactly_one_of_two_concurrent_restores
restore_object_usecase_completes_suspended_null_version_in_place
restore_object_usecase_reports_ongoing_conflict
rustfs-scanner::lifecycle_integration_test serial_tests::
test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore
Verification: the CI ILM lane filter now runs 53/53 green locally.
* chore: address review follow-ups on this branch
Four items from the adversarial review that were still open.
- Restore the assertion `test_bucket_replication_replayed_delete_marker_
preserves_source_mtime_without_source_restart` is named for. The branch
had replaced the backlog#867 mtime check with `assert_replication_
converged`, which any successful replication satisfies, and deleted the
two helpers it needed — so the regression the test exists to catch would
now pass. This matters here specifically because the branch changes the
flag feeding `replication_delete_remove_options` and routes replay
through a new file and ordering.
- Drop `read_config_no_lock_preserve_empty`: zero production callers (the
one real consumer calls the `_with_metadata` variant directly). Its test
stanza now exercises that variant, so the coverage moves to live code
rather than being deleted.
- Revert the `bytesize` bump. It is a no-op: `Cargo.lock` already pinned
2.7.0 before this branch and is untouched, so the caret range already
resolved there. Nothing in the diff uses the crate.
- Split the AGENTS.md "Adversarial Validation" policy change out of this
branch. The edit is defensible on its own, but it relaxes the review gate
that this branch has to pass, so it should land as its own PR reviewed on
its own merits rather than bundled with the change that benefits from it.
The reverted hunks are unchanged and ready to re-apply.
Not changed, deliberately: the missing-sidecar path still fails closed.
`missing_bucket_incarnation_sidecar_for_new_metadata_fails_closed` pins
that on purpose, and serving a non-authoritative Object Lock state would
be the wrong trade. The residual concern stands and is recorded in review
— a crash between the two writes in `persist_new_and_set` leaves the
bucket unloadable until DeleteBucket+CreateBucket, and the repair branches
in `migrate_legacy_metadata` and `make_bucket` are unreachable dead code
for that case. Resolving it needs the read path and the (transaction-lock
holding) repair path to be separated, which is more than a follow-up edit.
* test(ci): serialize the new bucket-incarnation tests
The five tests this branch adds around the incarnation / lifecycle fence
drive `init_bucket_metadata_sys` and `bucket_metadata_sys_of` — process-global
OnceLock state that `serial_test`'s `#[serial]` cannot protect across
nextest's process boundary — and they delete+recreate buckets, the shape that
raced into InsufficientWriteQuorum in backlog#937.
Add them to the `ecstore-serial-flaky` group in both the default and ci
profiles (nextest evaluates a named profile's own overrides list, so the
ci mirror is required). Preventive serialization only, no retries.
Not a full fix for the review comment: `bucket_delete_waits_for_config_
mutation_fence` still proves liveness with a fixed 200ms sleep plus
`assert!(!delete.is_finished())`. Turning that into readiness polling needs
a production-side signal to wait on — asserting "still blocked" is inherently
a negative. Serializing the group removes the parallel-load pressure that
makes the window fragile; the sleep itself is left for a follow-up.
* test(ecstore): pin that a drained bucket is actually deletable
`DeleteBucket`'s emptiness check is `has_xlmeta_files`, a raw scan of the
bucket directory on local disks — not an S3-level listing. So "the client
drained the bucket" and "the bucket is deletable" are two different
contracts, and only the first one was covered.
That gap is what the `S3 Implemented Tests` lane is failing on: 219 cases,
all `BucketNotEmpty` on `nuke_prefixed_buckets`, with every test body
passing. The first one is `test_versioning_obj_suspend_versions`, reported
by pytest as PASSED followed by ERROR at teardown.
Add the missing assertion for the unversioned path: PUT, client DELETE,
then assert no `xl.meta` survives and `DeleteBucket` succeeds. It passes —
which is itself a result: the plain delete path leaves no residue, so the
s3-tests failure is not there.
The versioning-suspended path is the remaining suspect (the client DELETE
leaves a null delete marker, and draining means purging it by
`versionId=null`). It is not covered here: `BucketVersioningSys` resolves
through the ambient `get_bucket_metadata_sys()` OnceLock, which this unit
env cannot set, so the bucket never actually reports as suspended. That
repro belongs at the e2e layer where a real server owns the versioning
state.
* fix(ecstore): let an explicit null-version delete purge its delete marker
Root cause of the `S3 Implemented Tests` lane: 219 cases, all
`BucketNotEmpty` on `nuke_prefixed_buckets`, every test body passing.
On a versioning-suspended bucket a client DELETE leaves a null delete
marker — correct S3 semantics, and an `xl.meta` on disk. Draining the
bucket therefore means purging that marker as `?versionId=null`, which is
what `nuke_bucket` does before `DeleteBucket`. That purge was rejected:
explicit null-version purge of the null delete marker must succeed,
got [Some(MethodNotAllowed)]
so the marker survived, and `DeleteBucket`'s emptiness check — a raw
`has_xlmeta_files` scan of the bucket directory, not an S3 listing — kept
reporting the bucket as non-empty.
The two sides of the version comparison in the batch delete loop are in
different namespaces. `goi.version_id` is the client-facing identity, where
`from_file_info` synthesizes `Some(Uuid::nil())` for a null version on a
versioned *or versioning-suspended* bucket. `version_id` is the storage
identity, where `delete_file_info_version_id` maps an explicit
`?versionId=null` to `None`. Comparing them raw makes the purge look like a
version mismatch, so `explicit_delete_marker` is false and the
`MethodNotAllowed` from the lookup is recorded as a delete failure.
This only became reachable on this branch: previously `check_opts` did not
carry `dobj.version_id`, so `set_disk_delete_creates_delete_marker` was
true, `object_lock_check_required` was false, and the lookup that produces
`MethodNotAllowed` never ran. Adding the version id to `check_opts` lit up
a comparison that was already wrong.
Normalize both sides through `delete_file_info_version_id`.
The regression test injects a real Suspended bucket-config snapshot — the
delete path reads versioned/suspended from that snapshot, not from `opts`,
so without it `from_file_info` never synthesizes the null version id and
the branch is not reached. Mutation-checked: restoring the raw comparison
fails the test with the exact `MethodNotAllowed` above.
* fix(app): drop the now-needless struct update
Reverting `crates/replication` to main removed the extra `MrfReplicateEntry`
fields, so this literal specifies every field again and `..Default::default()`
trips `clippy::needless_update` under `-D warnings`.
Caught by CI, not locally: I had run `cargo check --workspace --all-targets`,
which does not see clippy-only lints. Ran `cargo clippy --workspace
--all-targets -- -D warnings` here — clean.
* test(e2e): assert the fresh-volume classification
four_node_empty_legacy_volumes_start_as_fresh only started the cluster and
listed buckets — no assertion, so any classification path that still permits
startup left it green without proving the pre-created empty `.minio.sys`
directories were treated as fresh volumes.
Pin what that classification actually leaves behind: no buckets adopted into
the namespace, `.rustfs.sys/format.json` written on every drive, and the empty
legacy directory left untouched rather than migrated into.
* fix(bucket): apply the requested Object Lock to existing buckets
Site replication replays make-with-versioning against the destination,
carrying the source's `lockEnabled`. When the destination bucket already
exists it takes `force_create`, and the whole option-application block was
gated on `confirmed_missing` — so the call returned success while the replica
stayed unlocked. Replicated versions could then be deleted without the
retention the source enforces.
Object Lock enable is one-way, so applying it to an existing bucket is safe:
move it out of the creation-only gate, keeping `created` and versioning-only
options creation-scoped as before.
An existing authoritative bucket takes the `cache_bucket_metadata_in` branch,
which only caches, so the enable would have been dropped on restart. Persist
instead when the enable actually changed something.
Mutation-checked: restoring the creation-only gate fails the new
`force_create_enables_object_lock_on_an_existing_bucket` with "Object Lock
must be enabled on the existing bucket".
cargo nextest run -p rustfs-ecstore --lib: 3633 passed.
* fix(ecstore): box the generation-checked config mutation paths too
The earlier stack fix boxed `update` and `delete`, but an authorized
bucket-config mutation carrying an incarnation takes `update_if_incarnation`
/ `delete_if_incarnation` instead — which were still inlining the whole
resolve/load/save chain into an already-deep request future. Same overflow,
sibling path.
* fix(restore): keep the nil-version normalization the strip removed
Reverting the replication subsystem to main took `set_disk/replication.rs`
with it, but one line in that file was this branch's own fix rather than
replication work:
- self.version_id.filter(|v| !v.is_nil()) == fi.version_id.filter(|v| !v.is_nil())
+ self.version_id == fi.version_id
For a versioning-suspended object the expected version is `Some(Uuid::nil())`
while the read-back `FileInfo` carries `None`, so the raw compare reports
every suspended restore as "restored object changed before restore metadata
finalization" and the copy-back never commits. Same nil-vs-None mismatch as
the null delete-marker purge fixed earlier on this branch.
Caught by `Test and Lint (rio-v2)`, not by my local runs: the test lives in
`transition_commit_failure_tests`, gated behind `feature = "test-util"`, so
the 3633-test suite I had been running never included it. Re-ran with
`--features rio-v2,test-util`: 3722 passed.
* feat(replication): purge delete markers by the target's own version id
When a delete marker is replicated, the target assigns it a version id. The
purge that follows derived one from the *source* uuid instead, which is only
correct when the target mirrors source version ids. A generic S3 target does
not: the derived id addresses a version that does not exist there, so the
purge is a no-op and the replica keeps a marker the source has already
removed. Same failure class as #4401.
Record the id the target reports and address it directly on purge.
Data path, all of it driven by the object's internal metadata rather than the
`ReplicationState` wire form, which encodes positionally and cannot carry a
map:
- `rustfs-utils`: the `replication-delete-marker-version-<arn>` key family,
plus `strip_internal_prefix_preserving_case` — ARNs are case-sensitive and
the existing `strip_internal_prefix` lowercases.
- `ReplicationState` gains the map and a `..._corrupt` flag, both
`#[serde(skip)]`; `ReplicatedTargetInfo` carries the per-target id.
- `persist_target_delete_marker_versions` is merge-only. A delete arriving
over internode RPC has an empty map, so treating it as authoritative would
let a remote disk erase an id the local disk still holds.
- `delete_object_version` copies the map into `fi.metadata` before dispatch,
so the durable carrier crosses the wire even though the field does not.
- The keys are folded into the quorum hash through their normalized form:
the dual internal prefixes carrying one mapping share an identity, while a
genuine disagreement between disks still shows up as a quorum difference.
- `corrupt` (the prefixes disagreed) fails closed: skip the purge and warn
rather than guess an id and risk destroying a live version on the target.
Ported from the rc.1 branch, which cannot merge as a whole: its MRF replay
rewrite collides with #5659/#5671/#5672/#5673 and regressed
`MRF_PENDING_CAP`. main's MRF machinery is kept; only this capability moves
across. It touches no MRF code.
Two things did not survive the port, deliberately. The branch's
`missing_is_complete` purge regression does not exist here — it came from its
own HEAD-precheck rewrite, and main's simpler path never had it. And the
branch's `MrfReplicateEntry` ordering fields are MRF-redesign scope, left
behind.
Verification: cargo fmt --all --check, git diff --check,
cargo check --workspace --all-targets, and the suites for the four touched
crates — 4070 tests, 2 pre-existing failures unrelated to this change
(`system_resolver_negative_result_reaches_the_dns_allowlist`,
`test_resolve_domain_preserves_system_resolver_error_provenance`; both are
the sandbox DNS interception, they fail on a clean checkout too).
* fix(replication): keep the layer guard happy
scripts/check_architecture_migration_rules.sh matches on text, so the doc
comments naming `rustfs_filemeta::` read as a cross-layer dependency even
though nothing imports it. Reword them; the guard passes.
* fix(replication): make the target-version cap deterministic
Two defects in this PR, both found in review.
The cap was applied while iterating a `HashMap`, so *which* 1000 entries
survived depended on iteration order. Two disks decoding the same oversized
metadata could keep different subsets, hash differently, and lose quorum —
instead of both reporting the same corruption. Collect first, then truncate
in `BTreeMap` order, which is total and identical everywhere.
And `persist_target_delete_marker_versions` discarded the `corrupt` flag from
the RPC carrier, committing a delete-marker update that looked clean while the
exact remote marker identity was unknown. It now declines to merge a corrupt
carrier. Because the helper only ever inserts, declining leaves the durable
keys already on the object untouched, which is strictly safer than writing a
mapping we cannot trust.
Residual, stated rather than papered over: corruption confined to the RPC
carrier is not persisted as a sentinel, so a later reader of an object that
carried no durable keys still sees "legacy, no mapping" rather than "corrupt".
Persisting that would need a wire-format addition; the consumer already fails
closed on any corruption it can observe.
New test: `target_delete_marker_versions_cap_is_deterministic_across_decodes`
decodes the same 1050-entry map twice and asserts both the corrupt flag and
the retained subset agree.
* fix(replication): preserve multipart source mtime (#5669)
* fix(kms): repair unopenable ciphertext and cover the Vault backends (#5668)
* Add black-box behavior tests for KMS resilience and serialization
* fix(kms): repair unopenable ciphertext across backends
Black-box testing of the KMS crate surfaced several defects that make
encrypted data permanently unreadable.
Symmetric envelopes. The Local and Vault Transit backends returned raw
cipher output from `encrypt` while `decrypt` parsed a JSON envelope, so
anything sealed through the master-key path could never be opened again.
Local also discarded the AES-GCM nonce. Both now emit the same envelope
`decrypt` consumes, matching the Static backend.
Deterministic AAD. The object layer derived AEAD additional data by
serializing a `HashMap` directly. Iteration order differs per instance,
so a context rebuilt from storage produced different AAD bytes than the
one used to seal and the object stopped opening. Ordering by key removes
that dependency, matching the Static backend's existing `context_aad`.
Objects written with the default single-key context are unaffected,
since a one-entry map has only one serialization.
Cipher in the header projection. `metadata_to_headers` recorded the SSE
mode (`AES256` / `aws:kms`), which cannot represent ChaCha20-Poly1305,
so a ChaCha-sealed object came back claiming `aws:kms` and was opened
with the wrong cipher. The cipher now travels in
`x-rustfs-encryption-algorithm` — the header the storage layer already
reads but nothing ever wrote. Objects without it fall back as before.
Also: the Static backend ignored `key_spec` and always issued 256-bit
data keys; Local `list_keys` hardcoded `truncated: false`, ignored
`marker`, and paginated over unordered `read_dir`, so a paginating
client silently saw a partial key list; and Local and Vault KV2 reported
`key_id: "unknown"` from `decrypt` despite the envelope naming the
master key.
Co-Authored-By: Claude Opus 5 <[email protected]>
* test(kms): cover both Vault backends and key rotation
The behavior suite ran only against Local and Static, and its own harness
documented the gap: the Vault backends had no business-capability
coverage at all. Setting `RUSTFS_KMS_VAULT_TOKEN` now adds Vault KV2 and
Vault Transit to every `for_each_backend` spec against a live server.
That lane is what surfaced the Transit envelope defect fixed in the
previous commit.
`rotate` and `versioning` are advertised only by the Vault backends, so
until now every capability-gated branch for them took the
`UnsupportedCapability` side and the working half was never asserted — a
rotation that dropped prior key versions would have gone green. The new
`behavior_rotation.rs` pins that half: material sealed before a rotation
still opens after it, repeated rotations accumulate versions rather than
overwriting a single spare, and the history survives a restart.
Two test defects fixed. `objects_round_trip_across_sizes_and_algorithms`
asserted a 1-byte object differs from its own ciphertext, which collides
once every 256 runs; the assertion now applies only where a collision is
not realistic, and small objects stay covered by the tag check and the
decrypt round-trip. `test_from_env_selects_token_file` depended on
`RUSTFS_KMS_VAULT_TOKEN` being absent from the caller's environment and
now clears it explicitly.
The snapshots directory was also removed from `.gitignore`: insta
snapshots are the assertions themselves, so leaving them untracked gives
CI nothing to compare against. Only `.snap.new` scratch files are
ignored now.
Co-Authored-By: Claude Opus 5 <[email protected]>
* test(kms): adapt behavior suite to current key APIs
Rebasing onto main brought four API changes the suite predates.
`DeleteKeyRequest` gained `confirm_key_id`, and immediate deletion is now
gated on the server's `allow_immediate_deletion`. Scheduled deletions pass
`None`; the four specs that destroy a key outright echo the key id back
and opt the harness config in, which is what the gate asks of a real
caller.
`LocalBackupExportRequest` gained `sanitized_config`. These specs cover
the key-material path, so they seal no configuration and pass `None`.
`KmsCacheStats` became a named struct with real hit, miss, and eviction
counters. `cache_stats_returns_an_entry_count_and_no_hit_or_miss_data`
existed to pin the old placeholder behavior — that the second tuple
element was always zero — which main has since fixed, so it is now
`cache_stats_reports_hits_and_misses_separately` and asserts the counters
actually move.
Starting the service provisions the reserved probe key, so it shows up in
listings and backup bundles. Exact-set assertions filter it through a new
`without_probe_key` helper rather than naming it, keeping those specs
about the keys they seeded.
Co-Authored-By: Claude Opus 5 <[email protected]>
* fix(kms): bind the AAD to the stored context bytes
Review caught that canonicalizing the AAD on decrypt breaks objects sealed
before canonicalization existed, and it was right. The AAD is the
*serialization* of the encryption context, and `x-rustfs-encryption-context`
stores that exact byte sequence: `encrypt_object` fed one `HashMap` to the
AEAD and then moved the same map into the metadata the header is written
from, so the stored string is byte-identical to the AAD the object was
sealed under. Those objects are therefore recoverable — but only while
nothing round-trips the value through a `HashMap` and re-serializes it.
Recomputing sorted AAD on decrypt would have turned a readable object into
a permanently unreadable one. The previous behavior was worse than the
first analysis credited: it did not merely fail intermittently, it made
the failure deterministic.
`EncryptionMetadata` now carries `context_aad`, the bytes the object was
actually sealed with. Encryption records what it fed the AEAD, the header
projection stores those bytes verbatim (and preserves a legacy ordering
across a re-projection rather than rewriting it into sorted form), and
`headers_to_metadata` carries the stored string through untouched. Both
decrypt paths, SSE-KMS and SSE-C, prefer it and fall back to canonical
serialization only when no stored serialization exists. Canonicalization
still applies to everything newly sealed, so the original ordering bug
cannot recur.
Two tests pin this: a legacy record whose sealed bytes are non-canonical
must survive a full header round trip unchanged, and a context header
rewritten to an equivalent-but-reordered serialization must fail
authentication rather than silently re-deriving a working AAD. Both were
mutation-checked against the reinstated bug on each side.
Also from review: the lifecycle churn test asserted only that every
request was accounted for, which holds whether the state gate exists or
not, so both branches are now pinned deterministically after the churn
(asserting `refused > 0` on the concurrent phase would only trade the hole
for a scheduling flake). And the Local and Vault KV2 envelopes compare
`encryption_context` without authenticating it — `DekCrypto` seals only
the plaintext — which is now documented at both sites; closing it needs a
versioned envelope, since existing ciphertext was sealed without AAD.
Co-Authored-By: Claude Opus 5 <[email protected]>
---------
Co-authored-by: Claude Opus 5 <[email protected]>
---------
Co-authored-by: ccccpj <[email protected]>
Co-authored-by: 唐小鸭 <[email protected]>
Co-authored-by: Claude Opus 5 <[email protected]>
`delete_bucket_website` authorized through `s3:GetBucketPolicy` while
`put_bucket_website` used `s3:PutBucketPolicy`. The handler is a real
mutation — `rustfs/src/storage/ecfs.rs` calls
`delete_bucket_metadata_config(bucket, BUCKET_WEBSITE_CONFIG)`, permanently
removing the persisted website configuration.
So a principal holding only
{"Effect":"Allow","Action":["s3:GetBucketPolicy"],
"Resource":"arn:aws:s3:::victim"}
— an ordinary read-only "may read my bucket policy" grant — could send
`DELETE /victim?website` and destroy the configuration. On a bucket whose
policy grants that to `Principal: "*"`, it is reachable anonymously.
AWS treats this as its own permission: "This DELETE action requires the
S3:DeleteBucketWebsite permission." RustFS has no dedicated
`s3:PutBucketWebsite` / `s3:DeleteBucketWebsite` action, so this keeps the
existing bucket-config convention (`s3:PutBucketPolicy`, the same one
`put_bucket_request_payment` and `put_bucket_accelerate_configuration` use)
rather than adding actions, which would silently invalidate deployed
policies that already grant website writes.
Rather than correcting one constant, both handlers now route through a
single `bucket_website_config_authorize_action()`, so the read/write pair
cannot drift apart again.
Swept the rest of the surface while here: `delete_bucket_website` was the
only mutation handler authorizing through a Get*/List* action.
`delete_bucket_ownership_controls`, `put_bucket_ownership_controls` and
`put_bucket_metrics_configuration` return `Ok(())` with no authorization,
but none of them is implemented outside the access hook, so there is no
operation to authorize — left alone.
Adding a dedicated `s3:DeleteBucketWebsite` for full AWS parity is a
separate change with a policy-compatibility impact; noted, not done here.
Verification: cargo fmt --all --check, git diff --check,
cargo check -p rustfs --all-targets, cargo clippy -p rustfs --all-targets
(clean), and the new regression test. Mutation-checked: restoring
`GetBucketPolicyAction` turns
`bucket_website_config_never_authorizes_through_a_read_action` red.
The snapshots in crates/kms/src/api_types.rs pinned DeleteKeyResponse,
ListKeysResponse, DescribeKeyResponse and CancelKeyDeletionResponse, none
of which is serialized by any handler: those endpoints answer with
DeleteKmsKeyResponse and siblings in rustfs/src/admin/handlers/kms_keys.rs,
separate types carrying different fields. A breaking change to an admin
response could not fail them. Tag, untag and update-description had the
same gap, where the handler discards the kms-side response and serves its
own KmsKeyMetadataResponse.
Pin the shapes in the crate that produces them, and delete the four kms
mirrors. They were never in the pub use api_types list, had no
constructors and no callers, and only looked live because those snapshots
named them.
Keep the api_types snapshots that pin something real: configure, start,
stop and status are served verbatim by kms_dynamic, and the tag family
are live ObjectEncryptionService return types whose snapshots pin this
crate's public API rather than a wire shape.
A same-name CopyObject marks the operation `metadata_only`, which lets the
store layer rewrite `xl.meta` in place and leave the data blocks untouched.
The handler independently strips the source encryption metadata and calls
`sse_encryption`, which mints a *fresh* DEK. On an unversioned bucket both
happen at once, so the object ends up with a new DEK sitting beside ciphertext
sealed under the old one, and can never be decrypted again.
The mirror case is silent: an encrypted source copied without any destination
SSE keeps its ciphertext while losing the key metadata, so GET returns raw
ciphertext as if it were plaintext, with HTTP 200 and no error anywhere.
Keep `metadata_only` off whenever either side of the copy is encrypted, so the
store layer performs a full read/write rewrite through `put_object`. This is
the same resolution the versioned historical-restore path already uses for
this risk (issue #4238), and it matches MinIO's
`isSourceEncrypted || isTargetEncrypted -> metadataOnly = false` guard in
CopyObjectHandler.
The target half of the predicate deliberately tests `effective_sse` rather
than the request headers MinIO inspects: `effective_sse` also resolves the
bucket default-encryption rule, and `sse_encryption` mints a DEK from that
resolved value. A header-only check would miss a same-key copy performed under
a bucket default rule. The source half reuses `ObjectInfo::is_encrypted` so a
future encryption flavour is covered here as soon as it is recognised there.
Versioned buckets were already safe: that path falls through to `put_object`
regardless of `metadata_only`. RestoreObject also sets `metadata_only` but
only appends restore keys and never re-derives a DEK, so it is unaffected.
SizeSummary::tier_stats was populated for every scanned object but
apply_scanner_size_summary dropped it, so per-tier usage never reached
DataUsageInfo. Wire it through the same merge chain repl_target_stats
already uses, up to DataUsageInfo::tier_stats.
DataUsageEntry used the derived MessagePack encoding, which serialises
structs as arrays: appending a field turns the whole cache into a decode
error for older readers, so mixed-version nodes would invalidate each
other's cache every scan cycle. Give it the same hand-written
map-encoded Serialize DataUsageCacheInfo already carries, and record the
invariant in AGENTS.md.
Widen TierStats counters from i32 to u64 so a tier past 2^31 versions
cannot make checked_merge reject an entire usage snapshot, and drop the
duplicate TierStats/AllTierStats definitions in the scanner crate in
favour of the data-usage ones.
* feat(kms): add data key rewrap and wrapping inspection primitives
Rewrap re-protects an existing data key envelope with the master key's
current version without touching the data key itself, which is the
precondition for ever retiring an older version: until every envelope a
version wrapped has been moved off it, destroying that version orphans
every object whose data key it wrapped.
Adds KmsBackend::rewrap_data_key and its read-only counterpart
describe_data_key_wrapping, both gated by a new BackendCapabilities::rewrap
flag and defaulting to UnsupportedCapability. Vault KV2 unwraps with the
frozen version record that wrapped the envelope and re-wraps with the
current material; Vault Transit uses the native transit/rewrap endpoint so
the data key never enters this process.
No read or write path changes: nothing calls these yet.
* test(kms): cover the rewrap primitive against a scripted Vault
* fix(kms): resolve both key materials before the data key is unwrapped
Keeps every fallible step out of the window in which the plaintext data
key exists, so no error path can drop it without zeroizing it first.
* fix(kms): fail closed on Local key records this build cannot interpret
The Local backend's protection marker is the only version discriminator its
key records have, and three readers walked past it.
`ensure_missing_salt_can_be_generated` skipped every record it could not read
or parse, so a directory whose protection state is unknown still got a fresh
salt published before startup validation failed. That write is the
irreversible step: the next startup finds a salt file, never re-enters the
guard, and the evidence that the real salt was lost is gone. Every record the
guard now rejects already failed startup key validation a few lines later, so
no directory that initializes today stops initializing.
Backup export and restore folded an unknown marker into "material corrupt" /
"bundle corrupted". The record is intact and a newer build reads it fine, so
the operator response is a version change, not a disaster recovery. Both now
classify the marker before their schema parse, sharing one probe with the
backend reader.
`list_keys` dropped any record it could not decode from the page. Concurrent
removal stays a skip; anything else fails the listing rather than answering
"these are your keys" with a set that silently omits one.
* test(kms): cover every fail-closed path around the Local protection marker
Each test fails on the pre-fix code in the way the fix is about: the salt
cases because a replacement salt is published before startup validation
fails, the export and restore cases because the verdict comes back as
corruption, and the listing case because the record is edited out of the page.
The restore commit marker's unknown-version branch had no test at all,
unlike its Vault counterpart; it is now driven from the decoder, from the
restore entry point, and from backend startup.
Also states the widened salt guard in the Local backend operations doc,
including the operator recovery path for an unrecognized record.
* fix(kms): say 'not a readable JSON object' when the marker probe cannot parse
The probe now fails on any input that is not a JSON object, not only on
malformed JSON, so the message must cover both.
* test(kms): assert the salt file before the error variant
The replacement salt is written before the error the guard reports, so the
file assertion is the one that fails on a regression.
* test(kms): guard the new backup error variant's display string
* docs(kms): add the object-side bulk rekey job contract
Define the design contract for the bulk DEK re-wrap job before any of it is
built: work unit granularity, idempotency model, failure semantics, the
exclusion list, the metadata write constraints, and the ownership model.
The job re-wraps envelopes only and never rewrites object bodies, and it
never destroys a superseded key version: a half-finished run is a fully
serviceable state precisely because the old version still decrypts, so
folding destruction into the job would turn a resumable action into
irreversible loss on partial failure.
Records two positions that diverge from the originating request. Pause is
not provided; cancel plus cursor restart plus rate control covers what pause
is actually asked for, and none of the seven existing long-job frameworks
has a pause state. And the KEK version a given envelope was sealed under is
not observable today, from object metadata or from the decrypt response, so
recognizing the target state requires the re-wrap primitive to record a
version witness - stated here as the one interface both sides must agree on.
Refs rustfs/backlog#1642 (part of rustfs/backlog#1562)
* docs(kms): correct how the wrapping KEK version is recovered
The first draft claimed the wrapping KEK version was not observable at all,
and built the idempotent-skip and completion-evidence conclusions on that.
It is observable for every backend that rotates, so that framing was wrong.
The sealed blob under x-rustfs-encryption-key is the base64 of the backend
ciphertext, which is DataKeyEnvelope JSON; the read path in sse.rs already
discriminates on it via is_data_key_envelope. Vault KV2 records the version
in the envelope's master_key_version, and Transit leaves that field None on
purpose because its ciphertext self-describes as vault:vN:. Local and Static
hardcode None because neither rotates. Only AWS is genuinely opaque.
So the skip is achievable, and the honest statement is its cost: a base64
decode plus a JSON parse per scanned work unit, which belongs in the rate
budget. AWS becomes a scope-admission refusal instead of a silent
every-object rewrite on re-run.
Two traps replace the old over-broad claim. A bare None means three
different things across backends, so version extraction must be dispatched
by backend. And Local omits the version precisely because rotation is
rejected there, which couples it to backlog#1565: whichever change gives
Local a rotation history must start recording the version in the same
change, or Local joins AWS in the unreadable column.
The requirement left on the re-wrap primitive shrinks accordingly, from
"record a version" to exposing one backend-dispatched accessor and
reporting "already at target state" as a distinct outcome.
Refs rustfs/backlog#1642 (part of rustfs/backlog#1562)
api_types carried a second DeleteKeyRequest that nothing referenced: it is
absent from the lib.rs re-export list, so rustfs_kms::DeleteKeyRequest has
always resolved to types::DeleteKeyRequest through `pub use types::*`, and
the admin handler builds the real type via `use rustfs_kms::types::*`.
The copy had also drifted apart from the type it shadowed. It still called
force_immediate "for development/testing only" and described the 7-30 day
window as advisory, and it never gained the confirm_key_id field that the
immediate-deletion gate now requires. A caller that reached into api_types
and deserialized into it would silently drop confirm_key_id.
api_types::DeleteKeyResponse stays: it is live, pinned by the
kms_management_responses_have_stable_json_shapes snapshot alongside the
list/describe/cancel response shapes, and it mirrors the admin wire
response rather than duplicating types::DeleteKeyResponse, whose fields
differ. Its doc comment now records why no request twin sits beside it.
* docs: state that MinIO-encrypted objects are not readable by RustFS
Operators evaluating a MinIO migration had no warning that objects MinIO
wrote with SSE-S3, SSE-KMS, or SSE-C cannot be read back. The container
formats interoperate, so the limitation is easy to discover only after
the data has moved.
Document the limitation where a migration decision is actually made:
- minio-file-format-compat.md gains Part C, covering which object classes
transfer, the three seams that block each SSE mode with file:line
evidence, the reverse direction, and the current workarounds. It also
records that the `rio-v2` MinIO sealed-key parser does not close the
gap: the feature is absent from released artifacts, and the managed-SSE
detection gate is not feature-gated and returns before the parser runs.
- kms-backend-security.md gains an operator-facing warning next to the
backend comparison table, since configuring the static backend with
MinIO's key material looks like it should work and does not.
- s3-compatibility-matrix.md scopes its SSE row to RustFS's own
round-trip.
The read path treats an undetected MinIO-encrypted object as unencrypted
rather than failing, so all three notes tell operators to verify migrated
objects by content instead of by status code.
Refs rustfs/backlog#1638.
* docs: correct the failure mode for MinIO-encrypted objects
The read path does fail closed; the earlier text claimed ciphertext was
served as plaintext. MinIO's internal headers mark the object encrypted,
so the reader refuses when no material resolves. What is actually wrong
is the diagnosis: the refusal surfaces as a 500 InternalError.
docs(kms): correct the KV2 at-rest boundary and rotation rejection claims
The Vault KV2 section claimed the backend reports its confidentiality
boundary as `at_rest_protection: vault-kv2-acl` through `backend_info`.
That accessor, the `BackendInfo` type and its assertion test were removed
in rustfs/rustfs#5501, and no replacement reports the boundary. State
instead that the boundary is documented only, that `kms/status` exposes a
capability matrix covering supported operations rather than key-material
location, and that the backup manifest's `at_rest_protection` field is a
bundle declaration rather than a backend self-report.
The rotation section named `InvalidOperation` as the error Local and
Static return. Neither advertises `rotate`, so the product path falls to
the shared `KmsBackend` default and returns `UnsupportedCapability`;
`InvalidOperation` survives only in a test-only client helper.
fix(sse): strip inherited SSE key-id and algorithm on copy
strip_managed_encryption_metadata cleared the MinIO spellings of the
managed-SSE key id and seal algorithm but not their RustFS-native
counterparts, so a CopyObject destination kept the source object's
x-rustfs-encryption-key-id and x-rustfs-encryption-algorithm.
When the destination resolves to no server-side encryption, nothing
rewrites those keys. is_object_encryption_marker treats any remaining
x-rustfs-encryption-* key as proof the payload is encrypted, so the
plaintext destination reports ObjectInfo::is_encrypted, the reader takes
its encrypted branch, and the read fails closed with "encrypted object
metadata is incomplete" because the actual key material was stripped.
Add both constants to the strip list so a destination inherits no
encryption marker it has no material for.
* feat(kms): report configuration references that block a key deletion
Adds a KeyImpactReport that states which configuration still points at a
key, how exhaustively the sources were read, and which sources were not
consulted at all. The report deliberately carries no in-use or
safe-to-delete claim: it covers the configuration layer only, so an empty
reference list means nothing was found in the scanned sources, never that
the key is unreferenced.
Immediate deletion destroys key material without ever reaching the
deletion worker, so it never passed the worker's reference gate. The
manager now consults the same checker on that path and refuses with a
typed KeyStillReferenced error. This only ever adds a refusal; the
scheduled deletion path and the worker's blocking behaviour are
unchanged.
* test(kms): cover the immediate-deletion reference refusal
* feat(kms): surface configuration references on the admin key endpoints
DeleteKey and DescribeKey now return an impact section listing the
configuration that points at the key, so an operator scheduling a
deletion sees what will refuse to destroy the material instead of
learning it from a server-side log once the window has run out.
The section is reported, never acted on: scheduling still succeeds while
references exist, and the deletion worker's gate remains the only thing
that decides whether material is destroyed. An immediate deletion that
the manager refuses for an outstanding reference now answers 409.
* test(kms): pin the impact wire shape and the unreferenced force-delete path
* fix(kms): make the DescribeKey impact section opt-in
Collecting the section lists every bucket, and DescribeKey is polled, so
carrying that fan-out on the default read path trades a hot path's cost
for a diagnostic. It is now collected only for impact=true; without the
parameter the endpoint does exactly the work it did before and returns
no impact field.
A value that is neither true nor false is refused rather than read as
off, so a typo cannot answer a request for the section with a response
that merely lacks one. DeleteKey still reports unconditionally: that is
the request whose consequences the caller cannot otherwise see, and it
is not polled.
* fix(kms): box the query-parse refusal now that responses carry impact
The delete response grew an impact section, which pushed it past the
size clippy accepts inline in a Result. It is a full response body
rather than an error code, so it is boxed at the one place that returns
it as an error; the wire shape and the public field type are unchanged.
* fix(kms): persist and report the Vault KV2 key rotation timestamp
The rotation-age gauge reads KeyInfo::rotated_at and falls back to
created_at when it is absent. The KV2 backend never persisted a rotation
time and hardcoded None in describe_key, so a key rotated many times and
a key that was never rotated reported the same age.
Record the rotation time on the same check-and-set write that switches
the current version, and report the stored value from describe_key and
from the recovered-create path. Records written before the field existed
keep deserializing and stay unstamped: no timestamp is invented for a
rotation this node cannot vouch for.
* test(kms): cover Vault KV2 rotation timestamp persistence and legacy records
* docs(kms): correct the static backend's MinIO compatibility claim
`StaticConfig` claimed the backend derives DEKs via "HMAC-SHA256 +
AES-256-GCM, matching the MinIO builtin/static KMS wire format". Neither
half holds: the configured key is used directly as the AES-256-GCM key
with no derivation step, and wrapped DEKs are serialized as RustFS's own
`DataKeyEnvelope` JSON. MinIO's KMS ciphertext uses a different shape,
which `is_data_key_envelope` explicitly classifies as foreign (see the
`minio_legacy` case in encryption/dek.rs).
The claim as written tells a migrating operator that MinIO-written
ciphertext will open here, which it will not. Restate what the backend
actually does and point at rustfs/backlog#1638 for the real interop work.
Also refresh the neighbouring ciphertext-format note in static_kms.rs,
which still described a raw `ciphertext || nonce` layout that the JSON
envelope replaced.
* ci(minio-interop): fix the dead test selector and guard against empty runs
The job selected its tests with `-p rustfs-ecstore -E
'binary(minio_generated_read_test)'`. #5435 moved those reader tests from
crates/ecstore/tests/minio_generated_read_test.rs into the `rustfs` crate
as a `#[cfg(test)] mod`, which removed that test binary; the selector has
selected zero interop tests since. Point it at the tests where they now
live, verified locally:
cargo nextest list --run-ignored all -p rustfs --features rio-v2 \
-E 'test(minio_generated_read_test::)' # 4 tests, was 0
Add a guard step in front of the run. The old `binary(...)` form happened
to fail loudly once its binary disappeared, but the name-based form that
replaces it is a valid filterset even when it matches nothing, so a later
rename would silently reduce this job to a pass that asserts nothing. The
guard counts the selection and fails with an explicit reason; the count
comes from `filter-match.status`, since the JSON's top-level `test-count`
is the package total and ignores `-E`. `--no-tests=fail` on the run step
covers the same case if the guard is ever dropped.
Also record in the header what this job does and does not prove: MinIO
wrapped-DEK envelopes are still rejected by both envelope parsers, and
that work is tracked in rustfs/backlog#1638.
* fix(kms): page the Local key listing so the deletion sweep sees every key
The Local backend answered every ListKeys with the first `limit` entries of
`read_dir` and a hardcoded `truncated: false`, so the deletion sweep ended
after one page: on a deployment with more keys than a page, expired key
material past the first page was never destroyed, and the lifecycle gauges
published that partial page as if it were the whole key set.
Listing now orders the key set by identifier and pages through it, with the
marker as an exclusive lower bound on the identifier rather than an index, so
a key added or removed between pages — including the marker key, which the
sweep itself destroys — cannot make the listing skip keys or restart. Only the
page is read from disk, so a list costs the requested limit rather than the
size of the key set.
The pagination arithmetic lives in a shared helper so the other self-paging
backends can adopt the same semantics, and the sweep now stops instead of
re-listing when a backend hands back the cursor it was just given.
* fix(kms): give ListKeys a defined zero-limit and cursor contract
`GET /rustfs/admin/v3/kms/keys?limit=0` reached the Vault KV2 and Vault
Transit backends as a page size of zero, where the page arithmetic indexed the
element before an empty page and aborted the request. Both backends also
resolved the marker by searching for it in the key list, so a marker naming a
key that had since been removed silently restarted the listing from the
beginning instead of resuming after it.
All four self-paging backends now share one contract: a zero limit is answered
as an empty, non-truncated page without reaching the backend at all, and the
marker is an exclusive lower bound on the key identifier rather than a position
in the list. The Vault backends read metadata only for the page they return,
so a list costs the requested limit instead of the whole key set, and KV2 now
applies the usage and status filters it previously accepted and ignored.
* feat(kms): accept the AWS backend through KMS configuration
The AWS KMS backend could be constructed but not selected: the admin
configure API had no AWS variant and startup rejected the backend name.
The configure request pins the region rather than defaulting it, because
that configuration is persisted once and replayed on every node: leaving
the region to each node's ambient provider chain would let nodes address
different regions, and therefore different keys, while reporting an
identical configuration. The request accepts no credential fields, so
credentials stay with the aws-config provider chain on each node, and
`deny_unknown_fields` refuses attempts to submit them anyway.
* test(kms): cover AWS backend selection through the service manager
An end-to-end check that an admin configure request selects the AWS
backend, builds a client, and passes the startup health check. Marked
#[ignore]: it needs real AWS credentials, though it creates no key and
is therefore not billable on its own.
* test(kms): cover KV2 decrypt of pre-rotation envelopes offline
The forward half of the rotation contract - an envelope written before a
rotation still decrypts after it - was only exercised by the #[ignore]
live-Vault tests, so CI never verified it. The offline layer only had the
negative cases (a regressed version pointer must fail closed).
Drive encrypt -> rotate -> decrypt over the scripted Vault responder,
folding what each rotation writes back into the served state so the
material the decrypt resolves is the material the rotation persisted.
Also cover two consecutive rotations and pin that new envelopes carry the
rotated version.
* test(kms): cover Transit decrypt of pre-rotation data keys offline
Vault owns the transit crypto, so the offline responder cannot prove the
round trip - that stays in the #[ignore] live test. What it can pin is the
client-side wiring: after a rotation records the version bump, decrypting
a data key generated before it must forward the historical vault:v1:
ciphertext to Vault byte for byte and return the material Vault hands
back.
fix(admin): retire the query-string form of immediate KMS key deletion
Immediate deletion destroys master key material outright, and every
object encrypted under that key becomes permanently unreadable. The
delete endpoint accepted that request as a query parameter, which is the
form most easily issued by accident and the one that made the waiting
window bypassable.
The query string can now only schedule a deletion: `force_immediate`
with any value other than `false`, or a `confirm_key_id` parameter, is
refused with 400 rather than downgraded to a scheduled deletion, so a
caller cannot read the answer as "destroyed". The JSON body form is
unchanged and remains the single way to reach the service gate that
enforces the server opt-in and the echoed confirmation.
Classify the route accordingly: `RouteRiskLevel` gains `Critical` for
routes whose worst case is permanent loss of user data, and the KMS key
deletion route is the only member, pinned in both directions by a matrix
test. Endpoint-level coverage for the 7-30 day window bound is added for
every configured backend.
Refs rustfs/backlog#1585 (part of rustfs/backlog#1562)
* feat(kms): add a disaster-recovery drill harness for the Local backend
Rehearse the full backup/restore loop offline and return machine-readable
evidence: seed a sandbox deployment, seal sample objects through the
production encryption path, export a bundle, destroy the persistence layer,
preflight, restore, and decrypt every pre-disaster object again.
The evidence records the measured recovery point (one key is written past the
snapshot fence and must stay unrecoverable), the recovery time by phase, the
manifest digest before and after, and whether the restore treated its bundle
as read-only.
* test(kms): drill the Local disaster matrix and the interrupted cutover
Runs the harness against total key-directory loss, salt loss, and a torn key
record, asserting every pre-disaster object decrypts again while work past the
snapshot fence stays lost. Two further legs crash a restore exactly at its
commit point and prove the published marker names the bundle and the files it
still owes, then that re-running rolls forward and aborting rolls back.
The Vault leg needs a real server and is ignored by default: a Vault bundle
never carries the non-exportable Transit root, so what it drills is the
refusal to proceed before the operator has restored it natively.
* feat(kms): add an operator entry point for the disaster-recovery drill
Runs one rehearsal from environment configuration and writes the evidence
bundle, exiting non-zero on a failed verdict so a scheduled drill fails its
job instead of filing a bad report. It reads the same backup-KEK variables as
the admin backup API: drilling with the KEK real bundles are sealed under is
what proves that KEK is still retrievable.
* docs(kms): add the disaster-recovery drill runbook
Documents the procedure the harness automates: what a drill measures and why
the object probe rather than the manifest digest is the acceptance criterion,
the per-backend responsibility split, the disaster matrix, how to read the
evidence bundle, the two interrupted-cutover outcomes, and the Vault variant
whose cryptographic root comes back through Vault's own flow.
* chore(typos): accept RTO as a disaster-recovery term
* feat(policy): add built-in KMS role policies
KMSKeyAdministrator, KMSKeyUser and KMSAuditor ship as canned identity
policies so operators can express KMS role separation without hand-writing
the resource grammar. They grant only kms actions, so they compose with an
existing data-plane policy, and none of them confers kms:Configure,
kms:ServiceControl, kms:ClearCache, kms:Backup or kms:Restore.
* docs(kms): document per-key KMS authorization and the role templates
* test(kms): add an end-to-end negative authorization matrix
Covers the admin and SSE-KMS planes for a wrong identity, a wrong key, a
wrong action and an explicit Deny, each preceded by a positive control so a
denial cannot be an unpropagated policy. SSE-S3 and unencrypted objects are
asserted to stay exempt.
* test(replication): pin the SSE-KMS contract with per-key authorization on
The replication worker carries no request identity, so it must stay exempt
from SSE-KMS key authorization. Running the existing contract with the
switch enabled makes a regression in that exemption visible here.
* docs(kms): describe KMS configuration convergence as implemented
* docs(kms): document the landed KMS metric families and narrow the gaps list
* docs(kms): state that no request field sets the cache metrics switch
* docs(kms): correct the describe_key cache divergence bound
* feat(kms): add admin endpoints for key description and tag updates
Wire the KMS key metadata updates landed at the service layer to the admin
API: POST /v3/kms/keys/{update-description,tag,untag}. Each endpoint gates on
a dedicated KMS action scoped to the key its body names, records a handler-owned
audit entry for both the authorization denial and the outcome, and maps a
backend that cannot update key metadata to 501 rather than 404.
Refs rustfs/backlog#1586 (part of rustfs/backlog#1562)
* fix(admin): audit metadata attempts refused by an unavailable KMS
* test(admin): register the new KMS metadata routes in the matrix
EventName::mask() turns a leaf variant's discriminant `v` into
`1 << (v - 1)`, so the enum can hold at most 64 variants in total —
compound "All" variants included, since they consume discriminants
even though they own no bit and push every later leaf further up the
range. The last variant is KmsServiceStopped at 61, leaving 3 slots.
Past the budget, `1u64 << (v - 1)` shifts by 64 or more: debug builds
panic with a shift overflow, release builds silently mask the shift
amount down and hand back a bit that already belongs to another event.
Either failure surfaces far from the line that added the variant.
The existing test_mask_bit_budget_is_not_exhausted only bounds the
events listed in the test-local ALL_EVENT_NAMES, so a variant nobody
remembered to list went unchecked. Add three layers instead:
- A const assertion on LAST_EVENT_NAME_VALUE, anchored on the last
variant, that fails `cargo build` once the discriminant passes 64.
- An exhaustive-match tripwire next to ALL_EVENT_NAMES, so adding a
variant breaks compilation with a non-exhaustive-patterns error that
points at the budget notes.
- Tests pinning that discriminants stay dense and fully listed, that
every leaf mask is non-zero and owns a unique bit (catching the
release-mode wrapped-shift collision), and that the last variant
still lands inside the u64.
Document the budget and the three guards on mask() so the next person
adding an event sees the remaining headroom.
Refs: rustfs/backlog#1572
* feat(kms): add backup and restore admin API
Wires the merged KMS backup contract, Local export and Local restore into
the admin API: export a sealed bundle, run a zero-write restore preflight,
execute a confirmed restore, roll an interrupted restore back, and report
subsystem readiness.
- Dedicated kms:Backup / kms:Restore actions, recorded in the admin route
matrix. Neither is reachable through any other KMS action.
- Restore requires two independent confirmations: an echo of the bundle
manifest's backup id, and an explicitly named conflict policy (the
default never writes).
- The backup KEK comes from the environment and is refused when it reuses
a secret of the configured backend, compared both as the literal value
and as raw key bytes.
- No endpoint accepts a path: bundles are addressed by a validated name
under a configured root, and the restore target is always the server's
own configured key directory.
- Bundles now carry a sanitized configuration artifact built as an
allowlist projection, so a future backend credential field cannot leak
into a bundle by default. Restore verifies it and never applies it.
- Audit entries go through the existing KMS admin wiring and carry
identifiers only.
* test(kms): pin the backup admin API gates
Fixes the test KEK to a real 32-byte value and drives the export refusal
from the configured backend rather than from the handle that happens to
be available, so a Local handle cannot export on behalf of a backend
whose material RustFS does not own.
`VaultKeyData::baseline_version` pins the master key version that every
pre-versioning DEK envelope (one with no `master_key_version`) resolves to.
Builds released before versioned rotation do not know the field, and every KV2
lifecycle write — enable, disable, schedule/cancel deletion, tag, untag —
rewrites the whole record, so a single lifecycle call from an older node during
a rolling upgrade silently drops the baseline. Serde attributes cannot prevent
this: the code doing the dropping already shipped.
The loss is only latent until the next rotation. With the baseline gone,
rotation takes the first-rotation path again and freezes a *new* baseline at the
current version, so every legacy envelope permanently resolves to material that
never wrapped it. That is the point of no return, and it is the one this commit
blocks: rotation now lists the key's immutable version records first and refuses
when records exist while the record carries no baseline. Those two are created
by the same commit, so the combination can only mean the baseline was erased
afterwards. The refusal is decided from reads alone, before any write, and names
the version to restore — the oldest recorded version *is* the lost baseline,
since version records start at the baseline the first rotation froze.
Reads are diagnosed rather than blocked. A version-less envelope on a key with
no baseline resolves to the current version, which AES-256-GCM refuses to
unwrap when it is the wrong one, so no wrong plaintext can be returned. Only
after that failure does decrypt list the version records and re-report the
failure as the lost baseline. Refusing up front on the same evidence would break
reads that work today: an older node writes version-less envelopes wrapped with
whatever material is current, and those still decrypt. This also keeps the extra
listing off every read of pre-versioning data.
Self-healing (writing back `baseline_version = min(recorded)`) is deliberately
not done: during the mixed-version window that caused the loss, an older node
can erase it again on the next lifecycle call, so healing would mask an
unfinished upgrade instead of surfacing it. Moving the baseline to a KV path
older builds cannot rewrite is the real fix and is scheduled for GA.
Refs rustfs/backlog#1581 (part of rustfs/backlog#1562)
* fix(kms): honour the configured metadata cache TTL and metrics switch
KmsManager::new built the KmsCache from cache_config.max_keys alone, so
cache_config.ttl was dead configuration: every deployment ran the
hardcoded 300s window whatever the admin configure API was given, while
CacheSummary and the KMS config endpoint reported the configured value
back. cache_config.enable_metrics was never read anywhere.
Build the cache from the whole CacheConfig. The documented default is
reconciled down to the 300s the cache has always used rather than up to
the advertised 3600s, and now lives in one place (DEFAULT_CACHE_TTL)
instead of being duplicated across the four configure-request
converters, so the default path behaves exactly as before.
Behaviour change: a deployment configured through the admin API already
has ttl 3600 persisted, because the old converters wrote that default
into the stored config, so its describe_key staleness window widens from
an effective 300s to the 3600s it asked for. No cryptographic or
authorization path widens - encrypt, decrypt and generate_data_key go
straight to the backend and never read this cache. The Vault Transit
backend's own metadata cache, which does gate crypto through
ensure_key_state_allows, stays fixed at 300s and is now documented as
deliberately not operator-tunable.
The configured duration now reaches moka's builder, which panics above
1000 years, so CacheConfig::effective_ttl clamps to a 24h maximum the
way effective_timeout already clamps its own, and validate rejects a
zero TTL beside the existing max_keys check. Both config summaries and
the KMS config endpoint report the effective value, so the admin API
cannot advertise a lifetime the cache does not honour.
enable_metrics gates publication of the rustfs_kms_metadata_cache_*
families only; the counters behind the admin status API keep running
either way. No configure-request field sets it yet.
Refs rustfs/backlog#1584
* docs(kms): state why the Transit metadata TTL is not bound to the default
The comment claimed the constant matches config::DEFAULT_CACHE_TTL, which
reads as an invariant the code does not enforce. Say plainly that the
equality is a coincidence rather than a contract, and why binding the two
would be wrong: this cache gates crypto through ensure_key_state_allows,
so a later change to the operator-facing describe-cache default must not
be able to widen its staleness window.
Gate 2 in rustfs/backlog#1601 needs ten terminal Test and Lint samples at 3.
Push alone cannot supply them: this workflow cancels superseded runs on main,
and only 4 of the last 20 push-triggered Test and Lint jobs reached a terminal
state — 15 were cancelled. At that rate ten samples take roughly fifty merges,
so the experiment would sit open for days while the branch it measures keeps
moving.
The concurrency group is scoped by event_name, so a dispatched run has its own
group and is not cancelled by merge traffic. Including workflow_dispatch in the
raised value makes the sample collectable deliberately rather than by waiting
for pushes to survive.
PRs still get 2. The merge path is untouched.
Refs: rustfs/backlog#1598, rustfs/backlog#1601
The #5394 mitigation set it to 2 on the belief that three concurrent workspace
test links saturate the runner's overlay I/O. The sampler's cgroup v2 readings
show the pod has 14 CPUs and 28GB with a 2.1GB peak, so 2 throttles compilation
to a seventh of what is available and memory was never the constraint. The label
name sm-standard-4 had led everyone, including that mitigation and every
description written during this series, to assume four cores.
Raised to 3 for push events only. PRs keep 2, so the merge path is untouched
while the experiment runs.
Baseline over 17 non-cancelled samples at 2: median nextest/clippy step ratio
1.95, spread 1.85-2.06. Judging by that ratio rather than absolute wall clock is
what makes the signal usable — clippy runs on the same node at the same time and
is check-only for workspace members, so it never links the ~100 test binaries
this limit throttles, making it a control arm rather than a second measurement.
The criterion is the ratio dropping at least 10%, below about 1.76, with no 75m
timeout and no run showing three consecutive samples of rustc, collect2 or
rust-lld in D state. If it does not drop, the conclusion is that this limit is
not the bottleneck: fix it back at 2 and record the experiment. That is a
result, not a failure.
Kept at step level deliberately. rust-cache hashes CARGO/CC/CFLAGS/CXX/CMAKE/RUST
prefixed variables from process.env into its key, so promoting this to job level
would rotate every cache key on this lane.
Refs: rustfs/backlog#1598, rustfs/backlog#1601
The figures went only to $GITHUB_STEP_SUMMARY, which renders in the UI but
never appears in the job log — and the REST API exposes the log, not the
summary. So the one number the cache-all-crates decision rests on could not be
fetched by the script that needed it. Print to stdout as well, between markers,
and keep the summary rendering.
Cache Warm used one concurrency group for every trigger. GitHub keeps one
running plus one pending run per group, so a manually dispatched run sat as
pending until the next merge displaced and cancelled it — observed three times
in a row. That made the --timings gate in rustfs/backlog#1601 impossible to
trigger while main was busy, which is precisely when someone wants to measure.
Scoping the group by event lets the push and dispatch paths queue independently.
Neither gains cancel-in-progress, so a burst of merges still collapses into
"current run finishes, newest queued run follows".
The two paths can now overlap and race to save the same key. That is benign:
the loser finds the key already present and skips, and both builds produce the
same artifacts from the same commit.
* feat(health): drive KMS readiness from the probe status
Readiness reported the KMS ready on the service status bit alone, which says
the manager started, not that the backend can still serve a request. Consume
the background probe snapshot instead: a running service is withdrawn only
when a fresh snapshot shows failures at the threshold.
The readiness path stays free of backend calls — it reads the lock-free
snapshot — so a struggling KMS cannot be amplified into probe traffic of its
own. Everything the probe cannot speak to (no worker, no completed round,
a snapshot older than three probe intervals) leaves the previous status-bit
verdict standing, and the check remains off by default.
Refs rustfs/backlog#1584 (part of rustfs/backlog#1562)
* test(health): pin the readiness default at compile time
* feat(s3-types): add KMS service-control audit events
Configuration changes and service start/stop are management-plane actions
with no event name of their own, so they could not reach the audit
pipeline at all. Append three variants for them, following the existing
rule that KMS events are audit-only and live outside the `s3:` namespace,
so no bucket notification selector can expand to them.
* feat(admin): audit KMS management operations
Every KMS admin endpoint now builds an OperationContext from the
authenticated caller and hands it to the KMS layer, so the record the
manager already produces carries the principal, source address and
canonical request id instead of the internal placeholder.
A new adapter maps those records onto the server's existing AuditEntry
format and installs itself as the KMS audit sink at service assembly, so
KMS activity reaches the targets a deployment already operates. The
handlers emit directly for what the KMS layer cannot see: a request the
authorization gate rejects, and the endpoints with no context-aware KMS
entry point (data-key derivation and service control).
Only the failure class is recorded, never the error message, and the new
module joins the logging guardrail's checked files alongside the handlers
it serves.
The four consolidated ci keys plus the build keys still do not fit the
repository's fixed 10GB Actions cache quota, so LRU keeps evicting them:
measured demand is ci-dev 2331MB + ci-feat-proto 2310MB + ci-feat-rio 1951MB +
ci-uring 1317MB + two build legs at ~1429MB + cargo-deny 844MB, and main pushes
add two more build legs. That is roughly 14.4GB against 10.24GB. The symptom is
misleading: Cache Warm reports every job successful and the restores log "full
match: true", yet ci-feat-rio and ci-uring disappear from the cache list between
runs.
cache-all-crates was the wrong default for this repository. With it set to true,
rust-cache's cleanup returns before pruning ~/.cargo/registry/src and its config
archives the whole registry, so every cache carried the unpacked source tree of
every dependency — not, as the name suggests, just a few extra crates.
Setting it to false is rust-cache's own default and loses no coverage: the
package set comes from `cargo metadata --all-features`, a strict superset of any
single lane's feature closure; -sys crates are explicitly exempt from pruning,
since their source timestamps would otherwise trigger rebuilds; and everything
pruned is re-unpacked from the .crate files still in registry/cache, whose
mtimes crates.io normalises, so cargo fingerprints stay valid.
Applied to the setup composite and to audit.yml's own rust-cache. Cache Warm
now also reports the sizes of registry/src, registry/cache, registry/index,
~/.cargo/git and target/ to the step summary, immediately before rust-cache's
post step archives them, so the size of the effect is measured rather than
assumed.
The new sizes only appear once the cache key next rotates, since rust-cache
skips the save entirely on an exact key hit. Deliberately not forcing that by
bumping prefix-key: it would invalidate every family at once and produce a
repository-wide cold build.
Refs: rustfs/backlog#1598, rustfs/backlog#1600
