The snapshots in crates/kms/src/api_types.rs pinned DeleteKeyResponse,
ListKeysResponse, DescribeKeyResponse and CancelKeyDeletionResponse, none
of which is serialized by any handler: those endpoints answer with
DeleteKmsKeyResponse and siblings in rustfs/src/admin/handlers/kms_keys.rs,
separate types carrying different fields. A breaking change to an admin
response could not fail them. Tag, untag and update-description had the
same gap, where the handler discards the kms-side response and serves its
own KmsKeyMetadataResponse.
Pin the shapes in the crate that produces them, and delete the four kms
mirrors. They were never in the pub use api_types list, had no
constructors and no callers, and only looked live because those snapshots
named them.
Keep the api_types snapshots that pin something real: configure, start,
stop and status are served verbatim by kms_dynamic, and the tag family
are live ObjectEncryptionService return types whose snapshots pin this
crate's public API rather than a wire shape.
A same-name CopyObject marks the operation `metadata_only`, which lets the
store layer rewrite `xl.meta` in place and leave the data blocks untouched.
The handler independently strips the source encryption metadata and calls
`sse_encryption`, which mints a *fresh* DEK. On an unversioned bucket both
happen at once, so the object ends up with a new DEK sitting beside ciphertext
sealed under the old one, and can never be decrypted again.
The mirror case is silent: an encrypted source copied without any destination
SSE keeps its ciphertext while losing the key metadata, so GET returns raw
ciphertext as if it were plaintext, with HTTP 200 and no error anywhere.
Keep `metadata_only` off whenever either side of the copy is encrypted, so the
store layer performs a full read/write rewrite through `put_object`. This is
the same resolution the versioned historical-restore path already uses for
this risk (issue #4238), and it matches MinIO's
`isSourceEncrypted || isTargetEncrypted -> metadataOnly = false` guard in
CopyObjectHandler.
The target half of the predicate deliberately tests `effective_sse` rather
than the request headers MinIO inspects: `effective_sse` also resolves the
bucket default-encryption rule, and `sse_encryption` mints a DEK from that
resolved value. A header-only check would miss a same-key copy performed under
a bucket default rule. The source half reuses `ObjectInfo::is_encrypted` so a
future encryption flavour is covered here as soon as it is recognised there.
Versioned buckets were already safe: that path falls through to `put_object`
regardless of `metadata_only`. RestoreObject also sets `metadata_only` but
only appends restore keys and never re-derives a DEK, so it is unaffected.
SizeSummary::tier_stats was populated for every scanned object but
apply_scanner_size_summary dropped it, so per-tier usage never reached
DataUsageInfo. Wire it through the same merge chain repl_target_stats
already uses, up to DataUsageInfo::tier_stats.
DataUsageEntry used the derived MessagePack encoding, which serialises
structs as arrays: appending a field turns the whole cache into a decode
error for older readers, so mixed-version nodes would invalidate each
other's cache every scan cycle. Give it the same hand-written
map-encoded Serialize DataUsageCacheInfo already carries, and record the
invariant in AGENTS.md.
Widen TierStats counters from i32 to u64 so a tier past 2^31 versions
cannot make checked_merge reject an entire usage snapshot, and drop the
duplicate TierStats/AllTierStats definitions in the scanner crate in
favour of the data-usage ones.
* feat(kms): add data key rewrap and wrapping inspection primitives
Rewrap re-protects an existing data key envelope with the master key's
current version without touching the data key itself, which is the
precondition for ever retiring an older version: until every envelope a
version wrapped has been moved off it, destroying that version orphans
every object whose data key it wrapped.
Adds KmsBackend::rewrap_data_key and its read-only counterpart
describe_data_key_wrapping, both gated by a new BackendCapabilities::rewrap
flag and defaulting to UnsupportedCapability. Vault KV2 unwraps with the
frozen version record that wrapped the envelope and re-wraps with the
current material; Vault Transit uses the native transit/rewrap endpoint so
the data key never enters this process.
No read or write path changes: nothing calls these yet.
* test(kms): cover the rewrap primitive against a scripted Vault
* fix(kms): resolve both key materials before the data key is unwrapped
Keeps every fallible step out of the window in which the plaintext data
key exists, so no error path can drop it without zeroizing it first.
* fix(kms): fail closed on Local key records this build cannot interpret
The Local backend's protection marker is the only version discriminator its
key records have, and three readers walked past it.
`ensure_missing_salt_can_be_generated` skipped every record it could not read
or parse, so a directory whose protection state is unknown still got a fresh
salt published before startup validation failed. That write is the
irreversible step: the next startup finds a salt file, never re-enters the
guard, and the evidence that the real salt was lost is gone. Every record the
guard now rejects already failed startup key validation a few lines later, so
no directory that initializes today stops initializing.
Backup export and restore folded an unknown marker into "material corrupt" /
"bundle corrupted". The record is intact and a newer build reads it fine, so
the operator response is a version change, not a disaster recovery. Both now
classify the marker before their schema parse, sharing one probe with the
backend reader.
`list_keys` dropped any record it could not decode from the page. Concurrent
removal stays a skip; anything else fails the listing rather than answering
"these are your keys" with a set that silently omits one.
* test(kms): cover every fail-closed path around the Local protection marker
Each test fails on the pre-fix code in the way the fix is about: the salt
cases because a replacement salt is published before startup validation
fails, the export and restore cases because the verdict comes back as
corruption, and the listing case because the record is edited out of the page.
The restore commit marker's unknown-version branch had no test at all,
unlike its Vault counterpart; it is now driven from the decoder, from the
restore entry point, and from backend startup.
Also states the widened salt guard in the Local backend operations doc,
including the operator recovery path for an unrecognized record.
* fix(kms): say 'not a readable JSON object' when the marker probe cannot parse
The probe now fails on any input that is not a JSON object, not only on
malformed JSON, so the message must cover both.
* test(kms): assert the salt file before the error variant
The replacement salt is written before the error the guard reports, so the
file assertion is the one that fails on a regression.
* test(kms): guard the new backup error variant's display string
* docs(kms): add the object-side bulk rekey job contract
Define the design contract for the bulk DEK re-wrap job before any of it is
built: work unit granularity, idempotency model, failure semantics, the
exclusion list, the metadata write constraints, and the ownership model.
The job re-wraps envelopes only and never rewrites object bodies, and it
never destroys a superseded key version: a half-finished run is a fully
serviceable state precisely because the old version still decrypts, so
folding destruction into the job would turn a resumable action into
irreversible loss on partial failure.
Records two positions that diverge from the originating request. Pause is
not provided; cancel plus cursor restart plus rate control covers what pause
is actually asked for, and none of the seven existing long-job frameworks
has a pause state. And the KEK version a given envelope was sealed under is
not observable today, from object metadata or from the decrypt response, so
recognizing the target state requires the re-wrap primitive to record a
version witness - stated here as the one interface both sides must agree on.
Refs rustfs/backlog#1642 (part of rustfs/backlog#1562)
* docs(kms): correct how the wrapping KEK version is recovered
The first draft claimed the wrapping KEK version was not observable at all,
and built the idempotent-skip and completion-evidence conclusions on that.
It is observable for every backend that rotates, so that framing was wrong.
The sealed blob under x-rustfs-encryption-key is the base64 of the backend
ciphertext, which is DataKeyEnvelope JSON; the read path in sse.rs already
discriminates on it via is_data_key_envelope. Vault KV2 records the version
in the envelope's master_key_version, and Transit leaves that field None on
purpose because its ciphertext self-describes as vault:vN:. Local and Static
hardcode None because neither rotates. Only AWS is genuinely opaque.
So the skip is achievable, and the honest statement is its cost: a base64
decode plus a JSON parse per scanned work unit, which belongs in the rate
budget. AWS becomes a scope-admission refusal instead of a silent
every-object rewrite on re-run.
Two traps replace the old over-broad claim. A bare None means three
different things across backends, so version extraction must be dispatched
by backend. And Local omits the version precisely because rotation is
rejected there, which couples it to backlog#1565: whichever change gives
Local a rotation history must start recording the version in the same
change, or Local joins AWS in the unreadable column.
The requirement left on the re-wrap primitive shrinks accordingly, from
"record a version" to exposing one backend-dispatched accessor and
reporting "already at target state" as a distinct outcome.
Refs rustfs/backlog#1642 (part of rustfs/backlog#1562)
api_types carried a second DeleteKeyRequest that nothing referenced: it is
absent from the lib.rs re-export list, so rustfs_kms::DeleteKeyRequest has
always resolved to types::DeleteKeyRequest through `pub use types::*`, and
the admin handler builds the real type via `use rustfs_kms::types::*`.
The copy had also drifted apart from the type it shadowed. It still called
force_immediate "for development/testing only" and described the 7-30 day
window as advisory, and it never gained the confirm_key_id field that the
immediate-deletion gate now requires. A caller that reached into api_types
and deserialized into it would silently drop confirm_key_id.
api_types::DeleteKeyResponse stays: it is live, pinned by the
kms_management_responses_have_stable_json_shapes snapshot alongside the
list/describe/cancel response shapes, and it mirrors the admin wire
response rather than duplicating types::DeleteKeyResponse, whose fields
differ. Its doc comment now records why no request twin sits beside it.
* docs: state that MinIO-encrypted objects are not readable by RustFS
Operators evaluating a MinIO migration had no warning that objects MinIO
wrote with SSE-S3, SSE-KMS, or SSE-C cannot be read back. The container
formats interoperate, so the limitation is easy to discover only after
the data has moved.
Document the limitation where a migration decision is actually made:
- minio-file-format-compat.md gains Part C, covering which object classes
transfer, the three seams that block each SSE mode with file:line
evidence, the reverse direction, and the current workarounds. It also
records that the `rio-v2` MinIO sealed-key parser does not close the
gap: the feature is absent from released artifacts, and the managed-SSE
detection gate is not feature-gated and returns before the parser runs.
- kms-backend-security.md gains an operator-facing warning next to the
backend comparison table, since configuring the static backend with
MinIO's key material looks like it should work and does not.
- s3-compatibility-matrix.md scopes its SSE row to RustFS's own
round-trip.
The read path treats an undetected MinIO-encrypted object as unencrypted
rather than failing, so all three notes tell operators to verify migrated
objects by content instead of by status code.
Refs rustfs/backlog#1638.
* docs: correct the failure mode for MinIO-encrypted objects
The read path does fail closed; the earlier text claimed ciphertext was
served as plaintext. MinIO's internal headers mark the object encrypted,
so the reader refuses when no material resolves. What is actually wrong
is the diagnosis: the refusal surfaces as a 500 InternalError.
docs(kms): correct the KV2 at-rest boundary and rotation rejection claims
The Vault KV2 section claimed the backend reports its confidentiality
boundary as `at_rest_protection: vault-kv2-acl` through `backend_info`.
That accessor, the `BackendInfo` type and its assertion test were removed
in rustfs/rustfs#5501, and no replacement reports the boundary. State
instead that the boundary is documented only, that `kms/status` exposes a
capability matrix covering supported operations rather than key-material
location, and that the backup manifest's `at_rest_protection` field is a
bundle declaration rather than a backend self-report.
The rotation section named `InvalidOperation` as the error Local and
Static return. Neither advertises `rotate`, so the product path falls to
the shared `KmsBackend` default and returns `UnsupportedCapability`;
`InvalidOperation` survives only in a test-only client helper.
fix(sse): strip inherited SSE key-id and algorithm on copy
strip_managed_encryption_metadata cleared the MinIO spellings of the
managed-SSE key id and seal algorithm but not their RustFS-native
counterparts, so a CopyObject destination kept the source object's
x-rustfs-encryption-key-id and x-rustfs-encryption-algorithm.
When the destination resolves to no server-side encryption, nothing
rewrites those keys. is_object_encryption_marker treats any remaining
x-rustfs-encryption-* key as proof the payload is encrypted, so the
plaintext destination reports ObjectInfo::is_encrypted, the reader takes
its encrypted branch, and the read fails closed with "encrypted object
metadata is incomplete" because the actual key material was stripped.
Add both constants to the strip list so a destination inherits no
encryption marker it has no material for.
* feat(kms): report configuration references that block a key deletion
Adds a KeyImpactReport that states which configuration still points at a
key, how exhaustively the sources were read, and which sources were not
consulted at all. The report deliberately carries no in-use or
safe-to-delete claim: it covers the configuration layer only, so an empty
reference list means nothing was found in the scanned sources, never that
the key is unreferenced.
Immediate deletion destroys key material without ever reaching the
deletion worker, so it never passed the worker's reference gate. The
manager now consults the same checker on that path and refuses with a
typed KeyStillReferenced error. This only ever adds a refusal; the
scheduled deletion path and the worker's blocking behaviour are
unchanged.
* test(kms): cover the immediate-deletion reference refusal
* feat(kms): surface configuration references on the admin key endpoints
DeleteKey and DescribeKey now return an impact section listing the
configuration that points at the key, so an operator scheduling a
deletion sees what will refuse to destroy the material instead of
learning it from a server-side log once the window has run out.
The section is reported, never acted on: scheduling still succeeds while
references exist, and the deletion worker's gate remains the only thing
that decides whether material is destroyed. An immediate deletion that
the manager refuses for an outstanding reference now answers 409.
* test(kms): pin the impact wire shape and the unreferenced force-delete path
* fix(kms): make the DescribeKey impact section opt-in
Collecting the section lists every bucket, and DescribeKey is polled, so
carrying that fan-out on the default read path trades a hot path's cost
for a diagnostic. It is now collected only for impact=true; without the
parameter the endpoint does exactly the work it did before and returns
no impact field.
A value that is neither true nor false is refused rather than read as
off, so a typo cannot answer a request for the section with a response
that merely lacks one. DeleteKey still reports unconditionally: that is
the request whose consequences the caller cannot otherwise see, and it
is not polled.
* fix(kms): box the query-parse refusal now that responses carry impact
The delete response grew an impact section, which pushed it past the
size clippy accepts inline in a Result. It is a full response body
rather than an error code, so it is boxed at the one place that returns
it as an error; the wire shape and the public field type are unchanged.
* fix(kms): persist and report the Vault KV2 key rotation timestamp
The rotation-age gauge reads KeyInfo::rotated_at and falls back to
created_at when it is absent. The KV2 backend never persisted a rotation
time and hardcoded None in describe_key, so a key rotated many times and
a key that was never rotated reported the same age.
Record the rotation time on the same check-and-set write that switches
the current version, and report the stored value from describe_key and
from the recovered-create path. Records written before the field existed
keep deserializing and stay unstamped: no timestamp is invented for a
rotation this node cannot vouch for.
* test(kms): cover Vault KV2 rotation timestamp persistence and legacy records
* docs(kms): correct the static backend's MinIO compatibility claim
`StaticConfig` claimed the backend derives DEKs via "HMAC-SHA256 +
AES-256-GCM, matching the MinIO builtin/static KMS wire format". Neither
half holds: the configured key is used directly as the AES-256-GCM key
with no derivation step, and wrapped DEKs are serialized as RustFS's own
`DataKeyEnvelope` JSON. MinIO's KMS ciphertext uses a different shape,
which `is_data_key_envelope` explicitly classifies as foreign (see the
`minio_legacy` case in encryption/dek.rs).
The claim as written tells a migrating operator that MinIO-written
ciphertext will open here, which it will not. Restate what the backend
actually does and point at rustfs/backlog#1638 for the real interop work.
Also refresh the neighbouring ciphertext-format note in static_kms.rs,
which still described a raw `ciphertext || nonce` layout that the JSON
envelope replaced.
* ci(minio-interop): fix the dead test selector and guard against empty runs
The job selected its tests with `-p rustfs-ecstore -E
'binary(minio_generated_read_test)'`. #5435 moved those reader tests from
crates/ecstore/tests/minio_generated_read_test.rs into the `rustfs` crate
as a `#[cfg(test)] mod`, which removed that test binary; the selector has
selected zero interop tests since. Point it at the tests where they now
live, verified locally:
cargo nextest list --run-ignored all -p rustfs --features rio-v2 \
-E 'test(minio_generated_read_test::)' # 4 tests, was 0
Add a guard step in front of the run. The old `binary(...)` form happened
to fail loudly once its binary disappeared, but the name-based form that
replaces it is a valid filterset even when it matches nothing, so a later
rename would silently reduce this job to a pass that asserts nothing. The
guard counts the selection and fails with an explicit reason; the count
comes from `filter-match.status`, since the JSON's top-level `test-count`
is the package total and ignores `-E`. `--no-tests=fail` on the run step
covers the same case if the guard is ever dropped.
Also record in the header what this job does and does not prove: MinIO
wrapped-DEK envelopes are still rejected by both envelope parsers, and
that work is tracked in rustfs/backlog#1638.
* fix(kms): page the Local key listing so the deletion sweep sees every key
The Local backend answered every ListKeys with the first `limit` entries of
`read_dir` and a hardcoded `truncated: false`, so the deletion sweep ended
after one page: on a deployment with more keys than a page, expired key
material past the first page was never destroyed, and the lifecycle gauges
published that partial page as if it were the whole key set.
Listing now orders the key set by identifier and pages through it, with the
marker as an exclusive lower bound on the identifier rather than an index, so
a key added or removed between pages — including the marker key, which the
sweep itself destroys — cannot make the listing skip keys or restart. Only the
page is read from disk, so a list costs the requested limit rather than the
size of the key set.
The pagination arithmetic lives in a shared helper so the other self-paging
backends can adopt the same semantics, and the sweep now stops instead of
re-listing when a backend hands back the cursor it was just given.
* fix(kms): give ListKeys a defined zero-limit and cursor contract
`GET /rustfs/admin/v3/kms/keys?limit=0` reached the Vault KV2 and Vault
Transit backends as a page size of zero, where the page arithmetic indexed the
element before an empty page and aborted the request. Both backends also
resolved the marker by searching for it in the key list, so a marker naming a
key that had since been removed silently restarted the listing from the
beginning instead of resuming after it.
All four self-paging backends now share one contract: a zero limit is answered
as an empty, non-truncated page without reaching the backend at all, and the
marker is an exclusive lower bound on the key identifier rather than a position
in the list. The Vault backends read metadata only for the page they return,
so a list costs the requested limit instead of the whole key set, and KV2 now
applies the usage and status filters it previously accepted and ignored.
* feat(kms): accept the AWS backend through KMS configuration
The AWS KMS backend could be constructed but not selected: the admin
configure API had no AWS variant and startup rejected the backend name.
The configure request pins the region rather than defaulting it, because
that configuration is persisted once and replayed on every node: leaving
the region to each node's ambient provider chain would let nodes address
different regions, and therefore different keys, while reporting an
identical configuration. The request accepts no credential fields, so
credentials stay with the aws-config provider chain on each node, and
`deny_unknown_fields` refuses attempts to submit them anyway.
* test(kms): cover AWS backend selection through the service manager
An end-to-end check that an admin configure request selects the AWS
backend, builds a client, and passes the startup health check. Marked
#[ignore]: it needs real AWS credentials, though it creates no key and
is therefore not billable on its own.
* test(kms): cover KV2 decrypt of pre-rotation envelopes offline
The forward half of the rotation contract - an envelope written before a
rotation still decrypts after it - was only exercised by the #[ignore]
live-Vault tests, so CI never verified it. The offline layer only had the
negative cases (a regressed version pointer must fail closed).
Drive encrypt -> rotate -> decrypt over the scripted Vault responder,
folding what each rotation writes back into the served state so the
material the decrypt resolves is the material the rotation persisted.
Also cover two consecutive rotations and pin that new envelopes carry the
rotated version.
* test(kms): cover Transit decrypt of pre-rotation data keys offline
Vault owns the transit crypto, so the offline responder cannot prove the
round trip - that stays in the #[ignore] live test. What it can pin is the
client-side wiring: after a rotation records the version bump, decrypting
a data key generated before it must forward the historical vault:v1:
ciphertext to Vault byte for byte and return the material Vault hands
back.
fix(admin): retire the query-string form of immediate KMS key deletion
Immediate deletion destroys master key material outright, and every
object encrypted under that key becomes permanently unreadable. The
delete endpoint accepted that request as a query parameter, which is the
form most easily issued by accident and the one that made the waiting
window bypassable.
The query string can now only schedule a deletion: `force_immediate`
with any value other than `false`, or a `confirm_key_id` parameter, is
refused with 400 rather than downgraded to a scheduled deletion, so a
caller cannot read the answer as "destroyed". The JSON body form is
unchanged and remains the single way to reach the service gate that
enforces the server opt-in and the echoed confirmation.
Classify the route accordingly: `RouteRiskLevel` gains `Critical` for
routes whose worst case is permanent loss of user data, and the KMS key
deletion route is the only member, pinned in both directions by a matrix
test. Endpoint-level coverage for the 7-30 day window bound is added for
every configured backend.
Refs rustfs/backlog#1585 (part of rustfs/backlog#1562)
* feat(kms): add a disaster-recovery drill harness for the Local backend
Rehearse the full backup/restore loop offline and return machine-readable
evidence: seed a sandbox deployment, seal sample objects through the
production encryption path, export a bundle, destroy the persistence layer,
preflight, restore, and decrypt every pre-disaster object again.
The evidence records the measured recovery point (one key is written past the
snapshot fence and must stay unrecoverable), the recovery time by phase, the
manifest digest before and after, and whether the restore treated its bundle
as read-only.
* test(kms): drill the Local disaster matrix and the interrupted cutover
Runs the harness against total key-directory loss, salt loss, and a torn key
record, asserting every pre-disaster object decrypts again while work past the
snapshot fence stays lost. Two further legs crash a restore exactly at its
commit point and prove the published marker names the bundle and the files it
still owes, then that re-running rolls forward and aborting rolls back.
The Vault leg needs a real server and is ignored by default: a Vault bundle
never carries the non-exportable Transit root, so what it drills is the
refusal to proceed before the operator has restored it natively.
* feat(kms): add an operator entry point for the disaster-recovery drill
Runs one rehearsal from environment configuration and writes the evidence
bundle, exiting non-zero on a failed verdict so a scheduled drill fails its
job instead of filing a bad report. It reads the same backup-KEK variables as
the admin backup API: drilling with the KEK real bundles are sealed under is
what proves that KEK is still retrievable.
* docs(kms): add the disaster-recovery drill runbook
Documents the procedure the harness automates: what a drill measures and why
the object probe rather than the manifest digest is the acceptance criterion,
the per-backend responsibility split, the disaster matrix, how to read the
evidence bundle, the two interrupted-cutover outcomes, and the Vault variant
whose cryptographic root comes back through Vault's own flow.
* chore(typos): accept RTO as a disaster-recovery term
* feat(policy): add built-in KMS role policies
KMSKeyAdministrator, KMSKeyUser and KMSAuditor ship as canned identity
policies so operators can express KMS role separation without hand-writing
the resource grammar. They grant only kms actions, so they compose with an
existing data-plane policy, and none of them confers kms:Configure,
kms:ServiceControl, kms:ClearCache, kms:Backup or kms:Restore.
* docs(kms): document per-key KMS authorization and the role templates
* test(kms): add an end-to-end negative authorization matrix
Covers the admin and SSE-KMS planes for a wrong identity, a wrong key, a
wrong action and an explicit Deny, each preceded by a positive control so a
denial cannot be an unpropagated policy. SSE-S3 and unencrypted objects are
asserted to stay exempt.
* test(replication): pin the SSE-KMS contract with per-key authorization on
The replication worker carries no request identity, so it must stay exempt
from SSE-KMS key authorization. Running the existing contract with the
switch enabled makes a regression in that exemption visible here.
* docs(kms): describe KMS configuration convergence as implemented
* docs(kms): document the landed KMS metric families and narrow the gaps list
* docs(kms): state that no request field sets the cache metrics switch
* docs(kms): correct the describe_key cache divergence bound
* feat(kms): add admin endpoints for key description and tag updates
Wire the KMS key metadata updates landed at the service layer to the admin
API: POST /v3/kms/keys/{update-description,tag,untag}. Each endpoint gates on
a dedicated KMS action scoped to the key its body names, records a handler-owned
audit entry for both the authorization denial and the outcome, and maps a
backend that cannot update key metadata to 501 rather than 404.
Refs rustfs/backlog#1586 (part of rustfs/backlog#1562)
* fix(admin): audit metadata attempts refused by an unavailable KMS
* test(admin): register the new KMS metadata routes in the matrix
EventName::mask() turns a leaf variant's discriminant `v` into
`1 << (v - 1)`, so the enum can hold at most 64 variants in total —
compound "All" variants included, since they consume discriminants
even though they own no bit and push every later leaf further up the
range. The last variant is KmsServiceStopped at 61, leaving 3 slots.
Past the budget, `1u64 << (v - 1)` shifts by 64 or more: debug builds
panic with a shift overflow, release builds silently mask the shift
amount down and hand back a bit that already belongs to another event.
Either failure surfaces far from the line that added the variant.
The existing test_mask_bit_budget_is_not_exhausted only bounds the
events listed in the test-local ALL_EVENT_NAMES, so a variant nobody
remembered to list went unchecked. Add three layers instead:
- A const assertion on LAST_EVENT_NAME_VALUE, anchored on the last
variant, that fails `cargo build` once the discriminant passes 64.
- An exhaustive-match tripwire next to ALL_EVENT_NAMES, so adding a
variant breaks compilation with a non-exhaustive-patterns error that
points at the budget notes.
- Tests pinning that discriminants stay dense and fully listed, that
every leaf mask is non-zero and owns a unique bit (catching the
release-mode wrapped-shift collision), and that the last variant
still lands inside the u64.
Document the budget and the three guards on mask() so the next person
adding an event sees the remaining headroom.
Refs: rustfs/backlog#1572
* feat(kms): add backup and restore admin API
Wires the merged KMS backup contract, Local export and Local restore into
the admin API: export a sealed bundle, run a zero-write restore preflight,
execute a confirmed restore, roll an interrupted restore back, and report
subsystem readiness.
- Dedicated kms:Backup / kms:Restore actions, recorded in the admin route
matrix. Neither is reachable through any other KMS action.
- Restore requires two independent confirmations: an echo of the bundle
manifest's backup id, and an explicitly named conflict policy (the
default never writes).
- The backup KEK comes from the environment and is refused when it reuses
a secret of the configured backend, compared both as the literal value
and as raw key bytes.
- No endpoint accepts a path: bundles are addressed by a validated name
under a configured root, and the restore target is always the server's
own configured key directory.
- Bundles now carry a sanitized configuration artifact built as an
allowlist projection, so a future backend credential field cannot leak
into a bundle by default. Restore verifies it and never applies it.
- Audit entries go through the existing KMS admin wiring and carry
identifiers only.
* test(kms): pin the backup admin API gates
Fixes the test KEK to a real 32-byte value and drives the export refusal
from the configured backend rather than from the handle that happens to
be available, so a Local handle cannot export on behalf of a backend
whose material RustFS does not own.
`VaultKeyData::baseline_version` pins the master key version that every
pre-versioning DEK envelope (one with no `master_key_version`) resolves to.
Builds released before versioned rotation do not know the field, and every KV2
lifecycle write — enable, disable, schedule/cancel deletion, tag, untag —
rewrites the whole record, so a single lifecycle call from an older node during
a rolling upgrade silently drops the baseline. Serde attributes cannot prevent
this: the code doing the dropping already shipped.
The loss is only latent until the next rotation. With the baseline gone,
rotation takes the first-rotation path again and freezes a *new* baseline at the
current version, so every legacy envelope permanently resolves to material that
never wrapped it. That is the point of no return, and it is the one this commit
blocks: rotation now lists the key's immutable version records first and refuses
when records exist while the record carries no baseline. Those two are created
by the same commit, so the combination can only mean the baseline was erased
afterwards. The refusal is decided from reads alone, before any write, and names
the version to restore — the oldest recorded version *is* the lost baseline,
since version records start at the baseline the first rotation froze.
Reads are diagnosed rather than blocked. A version-less envelope on a key with
no baseline resolves to the current version, which AES-256-GCM refuses to
unwrap when it is the wrong one, so no wrong plaintext can be returned. Only
after that failure does decrypt list the version records and re-report the
failure as the lost baseline. Refusing up front on the same evidence would break
reads that work today: an older node writes version-less envelopes wrapped with
whatever material is current, and those still decrypt. This also keeps the extra
listing off every read of pre-versioning data.
Self-healing (writing back `baseline_version = min(recorded)`) is deliberately
not done: during the mixed-version window that caused the loss, an older node
can erase it again on the next lifecycle call, so healing would mask an
unfinished upgrade instead of surfacing it. Moving the baseline to a KV path
older builds cannot rewrite is the real fix and is scheduled for GA.
Refs rustfs/backlog#1581 (part of rustfs/backlog#1562)
* fix(kms): honour the configured metadata cache TTL and metrics switch
KmsManager::new built the KmsCache from cache_config.max_keys alone, so
cache_config.ttl was dead configuration: every deployment ran the
hardcoded 300s window whatever the admin configure API was given, while
CacheSummary and the KMS config endpoint reported the configured value
back. cache_config.enable_metrics was never read anywhere.
Build the cache from the whole CacheConfig. The documented default is
reconciled down to the 300s the cache has always used rather than up to
the advertised 3600s, and now lives in one place (DEFAULT_CACHE_TTL)
instead of being duplicated across the four configure-request
converters, so the default path behaves exactly as before.
Behaviour change: a deployment configured through the admin API already
has ttl 3600 persisted, because the old converters wrote that default
into the stored config, so its describe_key staleness window widens from
an effective 300s to the 3600s it asked for. No cryptographic or
authorization path widens - encrypt, decrypt and generate_data_key go
straight to the backend and never read this cache. The Vault Transit
backend's own metadata cache, which does gate crypto through
ensure_key_state_allows, stays fixed at 300s and is now documented as
deliberately not operator-tunable.
The configured duration now reaches moka's builder, which panics above
1000 years, so CacheConfig::effective_ttl clamps to a 24h maximum the
way effective_timeout already clamps its own, and validate rejects a
zero TTL beside the existing max_keys check. Both config summaries and
the KMS config endpoint report the effective value, so the admin API
cannot advertise a lifetime the cache does not honour.
enable_metrics gates publication of the rustfs_kms_metadata_cache_*
families only; the counters behind the admin status API keep running
either way. No configure-request field sets it yet.
Refs rustfs/backlog#1584
* docs(kms): state why the Transit metadata TTL is not bound to the default
The comment claimed the constant matches config::DEFAULT_CACHE_TTL, which
reads as an invariant the code does not enforce. Say plainly that the
equality is a coincidence rather than a contract, and why binding the two
would be wrong: this cache gates crypto through ensure_key_state_allows,
so a later change to the operator-facing describe-cache default must not
be able to widen its staleness window.
Gate 2 in rustfs/backlog#1601 needs ten terminal Test and Lint samples at 3.
Push alone cannot supply them: this workflow cancels superseded runs on main,
and only 4 of the last 20 push-triggered Test and Lint jobs reached a terminal
state — 15 were cancelled. At that rate ten samples take roughly fifty merges,
so the experiment would sit open for days while the branch it measures keeps
moving.
The concurrency group is scoped by event_name, so a dispatched run has its own
group and is not cancelled by merge traffic. Including workflow_dispatch in the
raised value makes the sample collectable deliberately rather than by waiting
for pushes to survive.
PRs still get 2. The merge path is untouched.
Refs: rustfs/backlog#1598, rustfs/backlog#1601
The #5394 mitigation set it to 2 on the belief that three concurrent workspace
test links saturate the runner's overlay I/O. The sampler's cgroup v2 readings
show the pod has 14 CPUs and 28GB with a 2.1GB peak, so 2 throttles compilation
to a seventh of what is available and memory was never the constraint. The label
name sm-standard-4 had led everyone, including that mitigation and every
description written during this series, to assume four cores.
Raised to 3 for push events only. PRs keep 2, so the merge path is untouched
while the experiment runs.
Baseline over 17 non-cancelled samples at 2: median nextest/clippy step ratio
1.95, spread 1.85-2.06. Judging by that ratio rather than absolute wall clock is
what makes the signal usable — clippy runs on the same node at the same time and
is check-only for workspace members, so it never links the ~100 test binaries
this limit throttles, making it a control arm rather than a second measurement.
The criterion is the ratio dropping at least 10%, below about 1.76, with no 75m
timeout and no run showing three consecutive samples of rustc, collect2 or
rust-lld in D state. If it does not drop, the conclusion is that this limit is
not the bottleneck: fix it back at 2 and record the experiment. That is a
result, not a failure.
Kept at step level deliberately. rust-cache hashes CARGO/CC/CFLAGS/CXX/CMAKE/RUST
prefixed variables from process.env into its key, so promoting this to job level
would rotate every cache key on this lane.
Refs: rustfs/backlog#1598, rustfs/backlog#1601
The figures went only to $GITHUB_STEP_SUMMARY, which renders in the UI but
never appears in the job log — and the REST API exposes the log, not the
summary. So the one number the cache-all-crates decision rests on could not be
fetched by the script that needed it. Print to stdout as well, between markers,
and keep the summary rendering.
Cache Warm used one concurrency group for every trigger. GitHub keeps one
running plus one pending run per group, so a manually dispatched run sat as
pending until the next merge displaced and cancelled it — observed three times
in a row. That made the --timings gate in rustfs/backlog#1601 impossible to
trigger while main was busy, which is precisely when someone wants to measure.
Scoping the group by event lets the push and dispatch paths queue independently.
Neither gains cancel-in-progress, so a burst of merges still collapses into
"current run finishes, newest queued run follows".
The two paths can now overlap and race to save the same key. That is benign:
the loser finds the key already present and skips, and both builds produce the
same artifacts from the same commit.
* feat(health): drive KMS readiness from the probe status
Readiness reported the KMS ready on the service status bit alone, which says
the manager started, not that the backend can still serve a request. Consume
the background probe snapshot instead: a running service is withdrawn only
when a fresh snapshot shows failures at the threshold.
The readiness path stays free of backend calls — it reads the lock-free
snapshot — so a struggling KMS cannot be amplified into probe traffic of its
own. Everything the probe cannot speak to (no worker, no completed round,
a snapshot older than three probe intervals) leaves the previous status-bit
verdict standing, and the check remains off by default.
Refs rustfs/backlog#1584 (part of rustfs/backlog#1562)
* test(health): pin the readiness default at compile time
* feat(s3-types): add KMS service-control audit events
Configuration changes and service start/stop are management-plane actions
with no event name of their own, so they could not reach the audit
pipeline at all. Append three variants for them, following the existing
rule that KMS events are audit-only and live outside the `s3:` namespace,
so no bucket notification selector can expand to them.
* feat(admin): audit KMS management operations
Every KMS admin endpoint now builds an OperationContext from the
authenticated caller and hands it to the KMS layer, so the record the
manager already produces carries the principal, source address and
canonical request id instead of the internal placeholder.
A new adapter maps those records onto the server's existing AuditEntry
format and installs itself as the KMS audit sink at service assembly, so
KMS activity reaches the targets a deployment already operates. The
handlers emit directly for what the KMS layer cannot see: a request the
authorization gate rejects, and the endpoints with no context-aware KMS
entry point (data-key derivation and service control).
Only the failure class is recorded, never the error message, and the new
module joins the logging guardrail's checked files alongside the handlers
it serves.
The four consolidated ci keys plus the build keys still do not fit the
repository's fixed 10GB Actions cache quota, so LRU keeps evicting them:
measured demand is ci-dev 2331MB + ci-feat-proto 2310MB + ci-feat-rio 1951MB +
ci-uring 1317MB + two build legs at ~1429MB + cargo-deny 844MB, and main pushes
add two more build legs. That is roughly 14.4GB against 10.24GB. The symptom is
misleading: Cache Warm reports every job successful and the restores log "full
match: true", yet ci-feat-rio and ci-uring disappear from the cache list between
runs.
cache-all-crates was the wrong default for this repository. With it set to true,
rust-cache's cleanup returns before pruning ~/.cargo/registry/src and its config
archives the whole registry, so every cache carried the unpacked source tree of
every dependency — not, as the name suggests, just a few extra crates.
Setting it to false is rust-cache's own default and loses no coverage: the
package set comes from `cargo metadata --all-features`, a strict superset of any
single lane's feature closure; -sys crates are explicitly exempt from pruning,
since their source timestamps would otherwise trigger rebuilds; and everything
pruned is re-unpacked from the .crate files still in registry/cache, whose
mtimes crates.io normalises, so cargo fingerprints stay valid.
Applied to the setup composite and to audit.yml's own rust-cache. Cache Warm
now also reports the sizes of registry/src, registry/cache, registry/index,
~/.cargo/git and target/ to the step summary, immediately before rust-cache's
post step archives them, so the size of the effect is measured rather than
assumed.
The new sizes only appear once the cache key next rotates, since rust-cache
skips the save entirely on an exact key hit. Deliberately not forcing that by
bumping prefix-key: it would invalidate every family at once and produce a
repository-wide cold build.
Refs: rustfs/backlog#1598, rustfs/backlog#1600
* feat(kms): observe key lifecycle from the deletion sweep
The sweep already pages through the whole key set, so the lifecycle
gauges come out of the pages it has in hand: no extra backend call is
made for them. It publishes the number of keys awaiting their deletion
deadline, the number of tombstones an interrupted removal left behind,
and how long ago the least recently rotated usable key was rotated
(counting from creation for keys that were never rotated), plus a
per-outcome counter of what the sweep acted on.
Every gauge is a label-less aggregate: a per-key label would carry key
identifiers into the metric stream and grow the series count with the
key set, so "this key is overdue for rotation" stays a threshold for an
alerting rule to apply to the aggregate. Keys the sweep destroys drop
out of the census, and a sweep that could not finish listing leaves the
gauges at their last complete values rather than understating them.
* feat(kms): expose Vault token TTL and fail-closed state as gauges
The renewal loop already tracks token expiry, so it now publishes the
seconds left on the active Vault token and whether the provider is
refusing to serve it. The fail-closed gauge re-evaluates the very gate
`VaultCredentialProvider::current` applies, so what operators see and
what the request path does cannot drift apart.
Both waits in the loop republish on a bounded cadence, so a scrape
landing between refresh cycles never reads a TTL frozen at the last
refresh or a fail-closed state that flipped after it. That costs a timer
and no Vault traffic, and the request path stays free of metric work.
Renewal successes and failures already land in the auth operation
counters, so nothing is double-counted here. Neither gauge carries a
label: the address, mount, auth path and token are all off limits as
label values, and there is one generation to describe.
* feat(kms): record real cache hit, miss and eviction metrics
The metadata cache reported (entry_count, 0) because moka exposes no hit
or miss counts, so the miss half of every cache report was a constant.
Track lookups and removals in the cache itself: hit/miss counters on the
lookup path, a moka eviction listener classifying removals by cause, and
an entry gauge refreshed whenever the entry set changes. The counters are
exported through the metrics facade under the rustfs_kms_ prefix with
static label values only, matching the operation-policy metrics, and are
also returned as a KmsCacheStats snapshot in place of the old tuple.
Cache semantics are unchanged: capacity, TTL and invalidation points are
the same, and remove now flushes pending maintenance so the gauge and the
removal notification describe the cache the caller sees.
Refs rustfs/backlog#1584
* fix(kms): report real cache counters through the admin status API
KmsStatusResponse.cache_stats mapped the old (entry_count, 0) tuple onto
hit_count and miss_count, so operators polling KMS status read the entry
count as a hit count and a miss count that was always zero.
Map the fields to the counters they claim to be, and add entry_count and
eviction_count as additive, defaulted fields so the entry number that
hit_count used to carry is still available.
Refs rustfs/backlog#1584
* fix(kms): refresh the cache entry gauge on lookup misses
The entry gauge was published only from the write paths, so an entry
dropped by TTL expiry left `rustfs_kms_metadata_cache_entries` reporting
a population that no longer existed until the next put, remove or clear.
A cache that goes quiet — entries ageing out with no further writes —
kept over-reporting indefinitely.
Republish the gauge from the lookup path when the lookup misses. A miss
is where expiry surfaces, and moka reaps expired entries in the
maintenance it runs during that same lookup, so the count read
afterwards reflects the reaping. Hits stay free of the extra work.
* docs(kms): correct the entry gauge convergence claim on the miss path
The comment on the miss-path gauge refresh said moka reaps expired
entries in the maintenance it runs on that same lookup. It does not:
`should_apply_reads` is gated on a full read log or an elapsed
housekeeping interval, so the removal that decrements `entry_count` and
reaches the eviction listener may land on a later lookup.
The behaviour and the test are unchanged — the gauge still converges,
and the test drives `run_pending_tasks` explicitly rather than riding on
that interval. Only the stated guarantee was wrong, so say interval
instead of same-lookup and record why forcing maintenance on the read
path was not the trade taken.
actions/checkout writes its token into .git/config as an http extraheader,
where it stays for the rest of the job. That matters more here than usual:
pull_request jobs run on self-hosted runners and execute the PR's own build.rs,
proc-macros and tests, any of which can read that file. Test and Lint holds
actions: write on top of that, so its token can cancel runs and delete the
Actions caches the whole pipeline now depends on — it was given
persist-credentials: false when that permission was added, and this extends the
same treatment to the other 45 checkouts.
Two are exempt because the token IS the credential the job needs.
helm-package's publish job pushes to rustfs/helm with it; clearing it would
break chart publishing. nix-flake-update is exempt pending verification: it
passes FLAKE_UPDATE_TOKEN to create-pull-request directly rather than reusing
.git/config, so it very likely does not need persistence, but that is unproven
and a broken weekly bot is not worth the guess. Both carry a
persist-credentials-exempt comment saying which.
scripts/security/check_persist_credentials.sh requires every checkout to either
clear its credentials or carry that comment, so the decision stays visible in
review rather than being an omission nobody notices.
Refs: rustfs/backlog#1598, rustfs/backlog#1602
sccache can only cache compilation units whose --emit includes link. That
covers workspace rlibs and nothing else: clippy is metadata-only, and the ~100
test binaries, the rustfs bin and every build script invoke the system linker.
So the headline claim that started this — s3select-query costing 17m13s inside
a 19m32s nextest build — does not by itself justify sccache, because cargo
prints "Compiling" when a crate starts and never when it finishes: that number
is the whole compilation tail, not one crate.
Adding --timings behind a workflow_dispatch input on Cache Warm answers it with
data instead. Read two shares off the report: workspace lib codegen against the
whole build, and s3select-query's own rlib. The plan in rustfs/backlog#1601
adopts sccache only above 50% and 25%; if linking dominates instead, the answer
is mold/lld plus split-debuginfo, which is precisely the region sccache cannot
reach.
Off by default: it doubles the ci-dev build, and this only needs running once
per question. Nothing about the normal warm path changes.
Refs: rustfs/backlog#1598, rustfs/backlog#1601
Two cleanups, neither changing what is built or published.
docker.yml loses its type=gha layer cache. The image build compiles nothing —
it downloads a release zip and runs apk/apt — so the cache could only save the
minute or two those take, against a real correctness problem: with
RELEASE=latest the binary URL is resolved by curl inside a RUN layer, and the
layer key does not include what it resolved to. A rebuild at the same RELEASE
value (a dispatch with version=latest, or a re-run of the same version) would
hit the old layer and ship the previous release's binary. mode=max also drew on
the same repo-wide 10GB Actions cache quota the Rust lanes are contending for.
Only RELEASE is passed as a build-arg now; it is the sole one the Dockerfiles
declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION and CHANNEL
were read by no stage, and BUILDTIME's $(date ...) was a literal string in the
YAML block rather than a substitution. The DOCKER_CHANNEL computation that fed
CHANNEL evaluated to "release" down every branch and is gone with it. BUILD_DATE
and VCS_REF stay unset even though the Dockerfiles declare them: supplying them
would change the published image labels.
The setup composite stops installing tools its callers do not use. protobuf-
compiler comes out of the apt list entirely — setup-protoc installs 34.1 into
the tool cache and prepends it to PATH, so the apt build was shadowed on every
run and never used; the same duplicate install is removed from the io_uring lane.
musl-tools, zip and unzip move behind install-build-packaging-tools, off for the
CI and coverage lanes and left on for build.yml, whose native musl leg needs
musl-gcc and whose packaging steps need zip. cargo-nextest and the
rustfmt/clippy components move behind install-test-tools, off only for build.yml,
which runs no tests and no lints; coverage and the nightly replication lane keep
them because both invoke nextest.
The action's github-token input is deleted along with its 20 call sites. Its
runs block never referenced it — setup-protoc uses github.token directly — so it
was 20 places handing a token to something that ignored it.
Refs: rustfs/backlog#1598, rustfs/backlog#1600, rustfs/backlog#1603
* docs(kms): record the cryptographic compliance position
RustFS links no FIPS-validated cryptographic module: the rustls provider is
the ordinary aws-lc-rs build, and every data-path AEAD is RustCrypto. The
crypto crate's default-on `fips` feature only selects PBKDF2+AES-GCM over
Argon2id, with the same RustCrypto implementations behind both branches, so
it cannot support a validation claim either.
Document that status, the terminology rules for external material, the real
semantics of the `fips` feature with a rename direction, the cost of the
three routes to a stronger position, and the sequencing rules for retiring an
algorithm.
Refs rustfs/backlog#1587 (part of rustfs/backlog#1562)
* docs(kms): document the mixed-version cluster constraints
Collect the cross-version constraints that landed with versioned rotation and
the check-and-set lifecycle work: which persisted formats decode both ways,
which guarantees only hold once every node is upgraded, how long nodes can
disagree on lifecycle state, and that reconfigure is persisted cluster-wide
but applied only on the node that handled it.
Adds the recommended rolling-upgrade sequence and the list of operations to
avoid while two builds are running.
Refs rustfs/backlog#1581 (part of rustfs/backlog#1562)
Two independent changes to the Test and Lint area.
The io_uring lane compiled 7 integration binaries to run none of their tests.
Job log 91309868055 shows the lib target reporting "18 passed; 3453 filtered
out" while every binary under crates/ecstore/tests/ reported "running 0 tests".
Adding --lib drops them from the build without changing the selected set.
The `uring_` filter itself must not be touched. libtest matches on substring, so
it also selects names containing `during_` — 6 of the 18 selected tests are such
incidental matches, and narrowing the filter to `io_uring` would silently drop
them. scripts/check_uring_lane_lib_only.sh asserts the precondition --lib
depends on: no test function whose name contains `uring_` may live under
crates/ecstore/tests/. A count floor would not do, because the dangerous case —
someone adding a matching test there — leaves the lib count unchanged and CI
green.
The resource sampler was measuring the wrong machine. The runners are ARC pods,
so /proc/loadavg, /proc/pressure/*, `free` and `df` are node-level and include
every other runner pod on the same Kubernetes node: one sample reported loadavg
6.67 with 832 threads node-wide while `ps` inside the pod showed about 10
processes. Judging a CARGO_BUILD_JOBS change on those numbers cannot work.
The sampler moves to scripts/ci/resource_sampler.sh and now records
/sys/fs/cgroup cpu.max, memory.max, memory.peak and the cpu/io/memory pressure
files, which are this pod's own. The node-level readings stay — co-tenancy is a
real cause of stalls, and a 9m57s plain `git checkout` was traced to it — but
are labelled NODE-LEVEL so nobody reads them as this job's load. Phase markers
are written on start, and clippy is now sampled too: it is the natural control
arm for a CARGO_BUILD_JOBS experiment, since --all-targets is check-only for
workspace members and never links the ~100 test binaries the limit throttles.
No numbers are changed in this commit. CARGO_BUILD_JOBS stays at 2 until there
is attributable data to change it on.
Refs: rustfs/backlog#1598, rustfs/backlog#1601
nix-flake-update opens a PR every Sunday that changes flake.lock and nothing
else. flake.lock is consumed only by Nix packaging — cargo never reads it — yet
it was in no paths filter, so each of those PRs ran the full Continuous
Integration pipeline and the merge then ran Build and Release too. Added to all
four lists that have to agree: ci.yml's push and pull_request paths-ignore,
build.yml's push paths-ignore, and ci-docs-only.yml's paths.
The cron stays. flake.lock still has consumers — anyone running `nix build` or
`nix develop` — so stopping the updates would remove the workflow's output, not
just its CI cost.
Those four lists drifting is a silent failure, so scripts/check_ci_paths_sync.sh
now asserts the pair that matters: ci.yml's pull_request paths-ignore must equal
ci-docs-only.yml's paths. An entry present only in the first means a PR touching
those files triggers neither workflow, nobody reports "Test and Lint" or "Quick
Checks", and the PR waits on a required check forever. It also asserts both
companion job names still exist, since renaming one produces the same hang. The
push list is not compared: no required check is reported for push events.
Also in this cleanup:
- nix-flake-update's GITHUB_TOKEN drops to contents: read. The branch push and
the pull request are both made by update-flake-lock with the
FLAKE_UPDATE_TOKEN PAT, so the write scopes were an unused repo-write
credential on an unattended weekly job.
- build.yml's build_docker dispatch input is documented as advisory. docker.yml
triggers on workflow_run and requires the triggering event to be a tag push,
so a manual dispatch never reaches it whatever this input says.
- The nine disabled workflow files get a banner saying so. Their
disabled_manually state lives in GitHub's UI and is invisible when reading the
file, which has already misled one audit into treating dead workflows as live.
Refs: rustfs/backlog#1598, rustfs/backlog#1603
A streaming GET holds snapshot leases on the object's data directories,
and DeleteObjects defers their physical cleanup until the leases are
released. A DeleteBucket issued inside that window passes the xl.meta
emptiness check but fails closed in the non-force delete_volume tree
removal on the leftover part files, returning BucketNotEmpty for a
logically empty bucket.
This is what intermittently failed the s3tests
test_encryption_sse_c_multipart_bad_download teardown in CI: the test
never reads its 30MiB GET body, so the server-side stream (and its
leases) stays alive until the connection drops, racing the teardown's
DeleteObjects + DeleteBucket sequence. With the body held open the
failure reproduces 5/5 locally; after this change it passes 20/20, and
the real s3-tests case passes 20 consecutive runs.
delete_volume now executes the registry-tracked pending deferred
deletions for the volume before removing the directory tree. Only data
dirs whose logical delete already committed are touched; unknown files
still fail closed with VolumeNotEmpty.
Three hardening changes with no effect on what any workflow produces.
Declare timeout-minutes on the 25 jobs that lacked it. GitHub's default is 360
minutes, and this repository has a history of runners stalling intermittently
(#5394) plus a measured 9m57s plain `git checkout` under node-level I/O
contention, so one wedged job could hold a runner for six hours out of a pool of
roughly 15-21. Budgets follow what the jobs actually do: 10 minutes for
echo-only and guard-script jobs, 30 for anything calling the GitHub API,
uploading release assets or pushing over the network.
scripts/security/check_job_timeouts.sh keeps it that way, checking only jobs
that declare runs-on so reusable-workflow callers are not flagged.
Pass workflow inputs and workflow_run fields through env instead of `${{ }}`
interpolation in run blocks. A git ref name may contain `$(...)` — any string
without a space is a legal tag — and interpolation pastes it into the script
where bash evaluates it. The worst instance was helm-package's final commit
message: it is built from the triggering tag name inside the job that holds the
cross-repository push token with rustfs/helm already checked out. Also converted
in build.yml, docker.yml and performance-ab.yml; the last is currently disabled,
but a disabled workflow can be re-enabled. Not touched: helm-package's
`contains(head_branch, '.')` tag test, since GitHub expressions have no regex
and this repository's tags carry no `v` prefix, so rewriting the condition would
change which builds publish a chart.
Give audit.yml a scheduled-failure alert and run it daily. A scheduled
cargo-deny failure usually means the dependency tree just matched a newly
published RustSec advisory — the most important signal this workflow produces,
and until now it was visible only to whoever happened to open the Actions tab.
coverage.yml and e2e-replication-nightly.yml already use this ci-8 mechanism.
The cron moves from weekly to daily so a new advisory against an unchanged tree
surfaces within a day instead of seven; the check list is untouched, since
splitting it into a light daily run and a weekly full run would create runs
where sources, bans and licenses go unverified.
Refs: rustfs/backlog#1598, rustfs/backlog#1602
#5532 consolidated ci.yml's nine cache sites into four keys with one writer
each, but the writers never run to completion: ci.yml cancels superseded runs
on main, and merges land far faster than its 70-minute pipeline. Measured over
15 consecutive main pushes — 12 cancelled, 2 failed, 0 succeeded. A cancelled
run never reaches rust-cache's post step (cache-on-failure does not cover
cancellation), so nothing was being saved and every PR paid a cold restore:
11.8-20.9 minutes of "Setup Rust environment" against 0.7-3.4 warm.
Move cache writing into its own workflow whose concurrency group does not
cancel in progress, and make every lane in ci.yml a pure reader. ci.yml keeps
cancelling superseded runs, which is correct — nobody needs test results for a
commit that is already three merges behind — while the caches still get
written.
Not simply disabling cancel-in-progress for main pushes in ci.yml: that would
run several full pipelines concurrently on a 15-21 runner self-hosted pool that
is already the bottleneck, which is the opposite of the goal. GitHub keeps at
most one running plus one pending run per concurrency group, so the new
workflow collapses a burst of merges into "current finishes, newest queued
follows" and occupies one runner at a time.
The warm builds are supersets of what the reading lanes compile, because a
reader restores only what the writer saved: --all-targets for the test binaries
nextest builds (including e2e_test, which test-and-lint excludes), plus the
e2e-test-hooks and rio-v2 feature combinations, whose different feature
resolution yields a different -Cmetadata. The ci-dev build carries the same
CARGO_BUILD_JOBS limit ci.yml puts on nextest, since it links the same ~100
test binaries and three concurrent links can wedge Cargo (#5394). ci-uring is
warmed on ubuntu-latest to match its reader: rust-cache's key covers runner.os
and arch but not the runner label or image.
Refs: rustfs/backlog#1598, rustfs/backlog#1600
The repository has 18 rust-cache families of 1.2-3.1GB each against GitHub's
fixed 10GB per-repo quota. Measured usage sat at 9.72GB, 9.96GB and 11.63GB on
three samples, so LRU eviction is continuous and the main lanes lose: ci-test
and ci-e2e were repeatedly absent from the surviving entries. That is what
makes "Setup Rust environment" bimodal — 0.7-3.4 minutes warm against
11.8-20.9 minutes cold.
Four changes, all cache-only. No job builds or tests anything different.
Collapse ci.yml's nine cache sites into four keys, each with exactly one
writer: ci-dev (test-and-lint writes; ILM, debug-binary, e2e-tests and e2e-full
read), ci-feat-rio (rio-v2 lint writes, its debug-binary reads), ci-feat-proto
(the swift leg writes for both protocol legs), and ci-uring, which stays alone.
e2e-full is easy to miss here: it already shared ci-e2e with e2e-tests and both
saved, so without an explicit 'false' it would have become a second unnamed
writer of ci-dev. rio/swift/sftp are deliberately NOT merged — measured at
2365/2307/1200MB they are not near-identical, and none is a superset of the
others.
Give each writer a superset warm-up on main. A writer's own steps are not
automatically a superset of its readers': clippy emits metadata only, the
nextest pass excludes e2e_test, and no lint lane enables e2e-test-hooks, whose
different feature resolution yields a different -Cmetadata. Without these
builds the readers would restore a cache missing precisely what they need.
Guarded to main, so the PR critical path is unaffected.
Stop writing tag-scoped caches in build.yml. A cache saved on refs/tags/X can
only be restored by a re-run of that same tag, so each release cycle wrote up
to 12 unreadable 1-2GB entries that evicted the hot lanes. Tag builds still
restore the main-scoped cache. The one real cost is that re-running a failed
leg of the same tag now falls back to main's cache.
Flip the composite action's cache-save-if default to 'false' and make the input
mandatory in practice. audit.yml was relying on the old "true" default: every
PR touching Cargo.toml or Cargo.lock saved a second, PR-scoped copy (~843MB
measured, job 91048468127) that pushed main-scoped lanes out of the quota. The
fail-safe direction is a cold cache, not a stolen quota slice.
scripts/security/check_cache_save_if.sh now asserts every call site states it,
wired into audit.yml next to the existing pin check.
While there, cargo-deny stops pulling the full setup composite. It compiles
nothing, so apt, protoc, flatc and nextest were pure overhead — but it does run
cargo metadata, and Cargo.toml pins datafusion and s3s as git dependencies that
must be materialised into ~/.cargo/git, so the cache itself stays.
Refs: rustfs/backlog#1598, rustfs/backlog#1600
* ci: stop running expensive jobs that cannot inform the result
Three independent fixes that all avoid burning self-hosted runners on work
whose outcome is already determined. None of them changes what is tested.
- ci-docs-only: add a "Quick Checks" companion job. It is a prerequisite for
gating ci.yml's expensive jobs behind quick-checks (rustfs/backlog#1599):
once "Quick Checks" is a required check, a docs-only PR would otherwise wait
on it forever. The steps are a byte-identical copy of ci.yml's quick-checks
rather than an `echo`, so that on a mixed PR the two same-named check runs
execute the same commands against the same merge ref and cannot disagree —
GitHub has no written contract for how it picks between same-named required
check runs, and the real job only takes 45-51s, leaving no timing margin to
rely on.
- ci: guard uring-integration with the same `closed` check every other job
already has. The pull_request trigger includes `closed` only so the
concurrency group cancels in-flight runs; this job had no guard and no
`needs`, so every closed or merged PR ran the full io_uring suite (4m17s,
7m19s and 7m31s on runs 30678272341, 30678117601 and 30662728539).
- ci: gate s3-lifecycle-behavior-tests on e2e-tests, matching
s3-implemented-tests. Both lanes only download the prebuilt debug binary, and
s3-implemented-tests already finishes later, so a green PR's wall clock is
unchanged; a red one stops holding a sm-standard-4 for up to 30 minutes.
Refs: rustfs/backlog#1598, rustfs/backlog#1599
* ci: gate the seven expensive jobs on Quick Checks
Every expensive job started in parallel with quick-checks, so a formatting or
architecture-guard failure still paid for the full pipeline. On run
30673292690 Quick Checks failed after 0.8 minutes and the run went on to burn
424.5 runner-minutes — 99.8% of it after the gate had already failed. The
self-hosted pool is 15-21 ARC runners and one full PR run needs about seven
sm-standard-4 concurrently, so those minutes come straight out of other PRs'
queue time (six runs measured 72-488 minutes queued).
quick-checks itself is compile-free and takes 45-51s, so a passing PR pays
about a minute of extra critical path.
REQUIRES the branch ruleset to list "Quick Checks" as a required check BEFORE
this merges. Adding `needs` gives these jobs a `skipped` conclusion for the
first time, and GitHub treats a skipped required check as satisfied — with
required_approving_review_count=0, a failing quick-checks would otherwise let
a broken PR merge. Ordering is tracked in rustfs/backlog#1599.
Refs: rustfs/backlog#1598, rustfs/backlog#1599
* ci: stop a PR run once Test and Lint has failed (#5530)
On run 30674613104 the e2e, ILM and sftp lanes had all failed while Test and
Lint and the rio-v2 variant kept running past 70 minutes. The run's verdict was
settled; the remaining lanes were spending sm-standard-4 time on a result
nobody could act on, and with one full PR run needing about seven of those
runners, that time comes out of other PRs' queue time.
Two mechanisms, both scoped to pull_request so main pushes, the merge queue and
the weekly schedule keep the full failure signal:
- test-and-lint-protocols: fail-fast on PRs, so one failing protocol leg stops
its sibling. This is the only part that also covers fork PRs, since it needs
no token.
- test-and-lint: on failure, cancel the run through the REST API.
Only test-and-lint may cancel. The lanes that are not required checks
(protocols, ILM, e2e, s3-tests) must never hold that power: a flake in one of
them would turn the required "Test and Lint" into `cancelled`, which blocks the
merge. A maintainer can merge today with sftp red, and that has to stay true.
The cancel step uses curl, not `gh`: every existing `gh` call in this repo runs
on ubuntu-latest, and the sm-standard-* images are custom and trimmed, so `gh`
is not known to exist there. Fork PRs are excluded by an explicit condition
rather than left to fail, since their GITHUB_TOKEN is forced read-only and
job-level permissions cannot raise it.
Job-level permissions must list contents: read alongside actions: write —
job-level permissions replace the workflow block instead of merging with it,
and dropping contents would break this job's checkout and the repo-token the
setup action passes to setup-protoc. Because that token can now cancel runs and
delete Actions caches, the checkout also sets persist-credentials: false so a
PR's own build.rs or proc-macro cannot read it back out of .git/config.
Refs: rustfs/backlog#1598, rustfs/backlog#1599
* ci: correct the companion-workflow comments
Addresses review feedback on #5528, which merged before these fixes were
pushed.
The ci-docs-only header claimed the ruleset already requires "Quick Checks".
It does not — that ruleset change is a separate step, and this file's whole
purpose is to land first so that change does not strand docs-only PRs. Say
what is true today.
Also move the byte-identical requirement onto ci.yml's quick-checks job, which
is the more likely edit site, instead of pointing at a comment that was not
there.
Three independent fixes that all avoid burning self-hosted runners on work
whose outcome is already determined. None of them changes what is tested.
- ci-docs-only: add a "Quick Checks" companion job. It is a prerequisite for
gating ci.yml's expensive jobs behind quick-checks (rustfs/backlog#1599):
once "Quick Checks" is a required check, a docs-only PR would otherwise wait
on it forever. The steps are a byte-identical copy of ci.yml's quick-checks
rather than an `echo`, so that on a mixed PR the two same-named check runs
execute the same commands against the same merge ref and cannot disagree —
GitHub has no written contract for how it picks between same-named required
check runs, and the real job only takes 45-51s, leaving no timing margin to
rely on.
- ci: guard uring-integration with the same `closed` check every other job
already has. The pull_request trigger includes `closed` only so the
concurrency group cancels in-flight runs; this job had no guard and no
`needs`, so every closed or merged PR ran the full io_uring suite (4m17s,
7m19s and 7m31s on runs 30678272341, 30678117601 and 30662728539).
- ci: gate s3-lifecycle-behavior-tests on e2e-tests, matching
s3-implemented-tests. Both lanes only download the prebuilt debug binary, and
s3-implemented-tests already finishes later, so a green PR's wall clock is
unchanged; a red one stops holding a sm-standard-4 for up to 30 minutes.
Refs: rustfs/backlog#1598, rustfs/backlog#1599
* feat(kms): add link_durably primitive and restore-marker startup guard
The local backend gains the two pieces the bundle restore path builds on:
a no-clobber hard-link publish primitive whose AlreadyExists case is
idempotent only for byte-identical content, and a fail-closed startup
guard that refuses to open a key directory holding a restore cutover
marker. The durable commit protocol and the key-id containment check
become pub(crate) so the restore module reuses them instead of copies.
* feat(kms): record a master-key verifier and pre-seal decrypt probe in export
Fill the manifest's master_key_verifier slot with an opaque one-way
value (scheme-prefixed, bound to the backup id and the KDF salt) so a
restore can detect a wrong operator-supplied master key before touching
any target state, and probe-decrypt every artifact as stored under the
backup KEK before the manifest may seal — digest equality alone only
proves the ciphertext landed intact. The payload decryption tail is
factored out and shared with the restore side so producer and consumer
cannot drift on the framing.
Also drops the stale KmsClient test import orphaned by the backend
refactor (#5501); the test suite did not compile without this.
* feat(kms): restore local backend key material from sealed backup bundles
The consumer side of the Local bundle export, as a four-phase protocol:
- Dry-run: full in-memory bundle decode (digest and AEAD verification of
every artifact), KDF-drift detection against the compiled-in
derivation, deployment and injected-generation checks (strictly lower
is rejected, equal stays allowed for repeated drills), master-key
verifier check, and target conflict enumeration - with zero writes.
- Staging: artifacts are committed durably into the .restore-staging/
subdirectory (invisible to the backend's key scan and orphan-temp
matcher) and every record is decryption-probed with the derived
master key both in memory before staging and again from the staged
bytes.
- Commit marker + cutover: the durably published .restore-commit.json
marker is the single commit point; cutover publishes staged files via
link_durably (salt first, keys after), then durably removes the
marker and drops staging.
- Crash re-entry: before the marker the target top level is untouched
and a re-run starts over; with the marker published, backend startup
fails closed and a re-run with the same bundle rolls forward while
abort_local_restore rolls back. Every interruption converges to the
complete old or complete new state.
Restore never goes through LocalKmsClient::new (which would mint a
fresh salt); the only write mode is the explicit
restore-into-empty-target policy, where an orphan salt or a foreign
marker already counts as non-empty. The bundle source stays strictly
read-only.
Refs rustfs/backlog#1572
Bring the dormant Resource::Kms variant to life: arn:aws:kms:::key/<key_id>
patterns (empty-account form, wildcards allowed in the id, alias/<name>
reserved as parse-only) now parse, validate, serialize back, and are matched
by KMS statements against the requested key id carried in Args::object.
Statements without KMS resources keep the legacy match-every-key behaviour,
as do call sites that pass no key resource, so nothing changes until the
admin/SSE authorization paths start passing key ids.
Statement validation rejects KMS resources on non-KMS statements, and bucket
policy validation rejects KMS actions and resources outright while stored
policies keep deserializing; evaluation skips pure-KMS bucket policy
statements with a warning. A kms:Decrypt action is added for the upcoming
SSE-KMS read path.
Refs rustfs/backlog#1582 (part of rustfs/backlog#1562)
* fix(kms): drop the KmsClient trait import removed by #5501
The local backup export tests (#5499) merged after #5501 folded the
KmsClient trait into KmsBackend, leaving a dead trait import that breaks
'cargo test -p rustfs-kms' compilation on main. create_key is an inherent
method on LocalKmsClient since #5501, so the import is unnecessary.
* fix(kms): CAS Transit metadata writes and bound the metadata cache
Transit KV metadata writes were whole-record overwrites with no
precondition, so two nodes mutating the same key could silently clobber
each other's lifecycle state, and the process-local metadata cache had
neither a TTL nor a capacity bound, so a key disabled or scheduled for
deletion on one node stayed usable on every other node until restart.
- Replace write_metadata_to_kv with a versioned read
(read_metadata_from_kv_versioned) plus a check-and-set write
(cas_write_metadata_to_kv); mutate_key_metadata re-reads the
authoritative record and re-runs the state gate on every attempt, and a
lost CAS race retries with a fresh snapshot (bounded budget) instead of
replaying the stale one.
- Migrate every read-modify-write caller: enable, disable, schedule and
cancel deletion, rotate version bump, the expired-key tombstone, and
both create paths (create-only CAS that read-confirms the winner on a
lost race).
- Bound the metadata cache with moka (300s TTL, 1024 entries) and drop a
key's entry when a transit data call reports it gone server-side.
- Fail closed when the synthesized-metadata fallback cannot be read or
persisted: the fabricated Enabled record is only served after a durable
create-only CAS write, closing the gate weakening documented as a KNOWN
RISK; the persistence fallback for pre-metadata keys is kept.
Refs rustfs/backlog#1581 (part of rustfs/backlog#1562)
* fix(kms): drop stale KmsClient trait import in local_export tests
The KmsClient trait was folded into KmsBackend (#5501), but the backup
export tests merged afterwards (#5499) still imported it, breaking the
crate's test build; create_key is an inherent LocalKmsClient method, so
the import is simply unused.
* fix(kms): make Vault KV2 lifecycle writes check-and-set
Every KV2 lifecycle write used to be a blind whole-record overwrite, so
two nodes racing on the same key could lose updates: a disable racing a
rotation wrote the pre-rotation record back (rolling back the version
and material of a committed rotation), concurrent same-name creates let
the later material win (orphaning DEKs wrapped under the earlier one),
and a cancellation racing the deletion sweep could be overwritten by
the tombstone (or resurrect an already tombstoned key).
All lifecycle mutations now go through a bounded check-and-set
read-modify-write loop: each attempt re-reads the record pinned to its
KV2 secret version, re-runs the state gate against the fresh snapshot,
and writes back check-and-set against exactly that version; after
LIFECYCLE_CAS_ATTEMPTS lost races the typed conflict error surfaces.
The loop composes with the operation policy's single-attempt rule for
non-idempotent writes: each write is still sent at most once, only the
whole read-gate-write cycle repeats. create_key becomes a create-only
write (cas=0) so exactly one of two concurrent creates commits and the
loser reports KeyAlreadyExists. The blind store_key_data primitive is
now test-only.
Reads and rotation additionally fail closed when the version history is
inconsistent: resolving material through a version record above the
current pointer is refused (that state only arises when a lost update
rolled back a committed rotation), and rotation refuses to extend a
history whose records reach more than one step past the current pointer
(one step ahead is the footprint of an interrupted rotation and still
recovers through the adopt path).
Refs rustfs/backlog#1581
Add a Grafana dashboard for the four KMS backend operation metrics
emitted at the policy choke point, Prometheus alert rules with
conservative default thresholds (pending staging baseline calibration),
and an operations runbook documenting the metric contract and the
response procedure for each alert. Register the new alert rules file in
the observability READMEs.
Refs rustfs/backlog#1584 (part of rustfs/backlog#1562)
The KmsClient trait was folded into KmsBackend (#5501), but the backup
export tests merged afterwards (#5499) still imported it, breaking the
crate's test build; create_key is an inherent LocalKmsClient method, so
the import is simply unused.
* feat(kms): export local backend key material as sealed backup bundles
Adds the producer side of the Local backup series on top of the #5483
contract: a directory-wide export fence gives the snapshot a single
consistent generation, every artifact is AEAD-wrapped under a
caller-supplied backup KEK that is separate from the business trust
hierarchy, and the sealed manifest with completeness marker is written
last so an interrupted export can never be mistaken for a restorable
bundle. Restore and the admin API land in follow-up changes.
* fix(kms): verify manifest digest against raw bytes, not re-serialized fields
The decode path recomputed the digest by re-serializing the parsed
manifest, which silently assumes every field's stored spelling survives
a parse-and-reprint round trip. Timestamps do not guarantee that: the
time zone annotation jiff emits depends on the host (IANA name, POSIX
TZ string, Etc/Unknown), and the legacy-compat parser rewrites
bracket-less spellings to +00:00[UTC]. On CI this made freshly written
bundles fail digest verification while passing locally.
Digest verification now operates on the raw stored bytes, normalized
only through the JSON value layer with the digest slot emptied in
place; parsed typed fields are never re-serialized on the decode path.
Sealing uses the same value-layer canonical form, and the export
additionally pins created_at to UTC so bundles are host-independent. A
regression test seals a manifest whose created_at spelling cannot
round-trip and proves decoding still verifies. One behavior sharpens:
inserting an explicit null reserved slot after sealing is now rejected
as a digest mismatch instead of being tolerated.
* fix(kms): make manifest digest canonicalization independent of map ordering
The canonical digest form serialized serde_json values directly, which
inherits the key order of serde_json's map type: sorted by default, but
insertion-ordered when any crate in the unified build graph enables the
preserve_order feature. The workspace-wide CI build unified that
feature while a per-crate local build did not, so the frozen fixture
digest matched in one environment and not the other — and a bundle
sealed by one build flavor would fail verification in the other.
Canonicalization now rebuilds every JSON object with bytewise-sorted
keys (array order preserved) before hashing, so the digest bytes are
identical regardless of feature unification. Reproduced by enabling
preserve_order in dev-dependencies (fixture test red), then verified
green with the fix under both map flavors.
* feat(kms): record operation metrics in the retry policy engine
Instrument policy::execute — the single choke point every outbound Vault
call and credential exchange already flows through — so no call site
needs its own instrumentation:
- rustfs_kms_backend_operations_total (counter): operation, op_class,
outcome (success / fatal / budget_exhausted / deadline_exceeded /
cancelled)
- rustfs_kms_backend_attempt_failures_total (counter): operation,
error_class (retryable_conn / retryable_status / fatal /
attempt_timeout)
- rustfs_kms_backend_operation_duration_seconds (histogram): wall-clock
duration including retries and backoff
- rustfs_kms_backend_operation_attempts (histogram): attempts used
Metric labels carry only static enum values (operation names, classes,
outcomes) — never key identifiers, key material, ciphertext, or tokens.
Emission goes through the process-global metrics facade recorder, the
same pattern the rest of the workspace uses, so no new wiring is needed
in rustfs/src.
Tests drive a paused-clock runtime under a thread-local debugging
recorder, so counts, attempts, and even the recorded (virtual-clock)
durations are asserted deterministically with zero real sleeps.
Refs rustfs/backlog#1569 (part of rustfs/backlog#1562)
* test(kms): add Vault fault-injection matrix
Offline cases inject transport faults locally and are fully
deterministic: a refused connection is retried up to the configured
budget, and a stalled connection is cut off by the per-attempt timeout
instead of hanging. Ignored cases run against a real dev Vault
(RUSTFS_KMS_VAULT_ADDR) and pin the fail-closed auth behavior: an
invalid token and a missing key each resolve in exactly one attempt.
Every case asserts through the policy metrics recorded by a
thread-local debugging recorder, which doubles as the request-count
assertion even against a real server. Throttling and recoverable 5xx
responses cannot be forced on a stock dev Vault; those paths stay
pinned by the scripted-Vault wiring tests and the engine tests.
Refs rustfs/backlog#1569 (part of rustfs/backlog#1562)
* feat(kms): add key lifecycle operations to the backend contract
Add enable_key/disable_key/rotate_key to KmsBackend with conservative
defaults returning the typed UnsupportedCapability error, mirroring
remove_expired_key. KmsManager gains matching pass-through methods and
drops cached key metadata after every successful state mutation so the
next describe observes backend truth. The local backend overrides
enable/disable, delegating to its state-machine-gated client methods;
rotation stays rejected, matching its advertised capabilities. New
dedicated policy actions kms:EnableKey and kms:DisableKey complete the
KMS action taxonomy alongside the existing kms:RotateKey.
* feat(admin): add KMS key enable/disable/rotate endpoints
POST /v3/kms/keys/enable, /v3/kms/keys/disable and /v3/kms/keys/rotate,
following the existing /v3/kms/keys handler conventions: key_id body
with keyId query fallback, {success, message, key_id, key_metadata}
responses, and 503 JSON while the KMS service is absent. Error mapping
keeps InvalidOperation/ValidationError at 400 like the sibling handlers
and surfaces UnsupportedCapability as 501 so a backend capability gap is
never mistaken for a missing key. Existing /v3/kms/keys handlers are
untouched apart from a visibility change on a private query helper.
* feat(kms): gate scheduled key deletion on bucket encryption references
Implement the DeletionReferenceChecker seam left by the deletion worker:
before any material is destroyed, every bucket's SSE configuration is
checked for a default KMS key reference and a hit blocks the removal.
The gate fails closed - an unpublished object store, a failed bucket
listing or an unreadable per-bucket encryption config all report a
blocking reference - because destroying key material is irreversible
while a blocked removal is simply retried on the next sweep. Registered
during init_kms_system before the service can start, so every worker
spawn observes it. Storage access goes through a new kms section of the
root storage facade.
* test(kms): add a scripted loopback Vault for policy wiring tests
A minimal HTTP/1.1 responder that serves canned Vault responses in order
and records the method/path sequence, so wiring tests can assert exactly
how many requests a code path performed (retries, read-confirm) without
a live Vault server.
* feat(kms): route Vault operations through the retry policy engine
Wire every outbound vaultrs call in the KV2 and Transit backends through
policy::execute, completing the wiring half of the operation policy work
(the engine landed separately):
- Reads (KV2 read/read_metadata/read_version/list, transit read/list/
encrypt/decrypt, health checks) run as ReadIdempotent: bounded retries
with exponential backoff and jitter on 429, recoverable 5xx, and
connection-level failures; 400/401/403/404 stay fatal.
- Writes (KV2 set/CAS set/delete_metadata, transit create/update/rotate/
delete, metadata writes) run as MutatingNonIdempotent: exactly one
attempt under the per-attempt timeout, never replayed. CAS conflicts
in the rotation protocol pass through unchanged as the concurrency
signal they are.
- Each attempt takes a fresh credential snapshot, so a retry after a
credential rotation uses the new token.
- Read-confirm recovery for lost create responses: when a create finds
an existing key that is exactly what it would have produced (same
algorithm, enabled, usable material, and for request-level creates the
same usage/description/tags), it reports the stored key as the create
result instead of KeyAlreadyExists. Any divergence keeps failing.
- Deletes treat already-deleted records as completed deletes (KV2
version records; transit metadata already did), so re-running an
interrupted deletion converges.
- A failed existence pre-check inside create now fails the create
instead of falling through to a blind overwrite (fail closed).
- The policy module sheds its allow(dead_code) now that it is wired.
Wiring tests run against a scripted loopback Vault and assert request
counts and endpoints for the retry, single-attempt, CAS-conflict, and
read-confirm paths.
Refs rustfs/backlog#1569 (part of rustfs/backlog#1562)
* feat(kms): enforce shared key state machine across backends
Unify the key state x operation matrix behind a single gate in
backends/mod.rs and wire it into the Local, Vault KV2 and Vault Transit
backends: Disabled keys reject encryption, data key generation and
rotation while still allowing decryption and lifecycle recovery;
PendingDeletion keys reject everything except decryption and
cancellation (including repeated deletion scheduling); cancellation now
requires an actual pending deletion everywhere. This closes the missing
gates on KV2 encrypt/generate and Local generate_data_key, and stops
enable_key from silently reverting a pending deletion.
Decryption is deliberately left ungated in Disabled/PendingDeletion — an
explicit, documented and tested deviation from AWS KMS, since gating it
would break reads of existing objects the moment a key is disabled.
Add shared contract tests driving the full matrix offline for Local (and
via ignored tests against a live Vault for KV2/Transit), a stateless
contract for Static, an SSE-shaped regression proving existing envelopes
stay decryptable after disable, and a pin on the known-risk Enabled
default of Transit's synthesized metadata fallback.
Refs rustfs/backlog#1571 (part of rustfs/backlog#1562)
* feat(kms): persist deletion deadlines and run a restartable deletion worker (#5491)
* feat(kms): add AppRole configuration surface for Vault auth
Extend VaultAuthMethod::AppRole with secret_id_file (re-read on every
login so external rotation is picked up), a configurable auth mount
(default "approle"), and an optional fail-closed safety window. All new
fields are serde(default) so previously persisted configurations keep
deserializing, and the strict admin-configure deserializer accepts them
as optional.
Environment selection: setting RUSTFS_KMS_VAULT_APPROLE_ROLE_ID switches
both Vault backends to AppRole; the secret_id comes from
RUSTFS_KMS_VAULT_APPROLE_SECRET_ID_FILE (path stored, file wins) or
RUSTFS_KMS_VAULT_APPROLE_SECRET_ID, following the static secret-key file
precedent. validate() rejects AppRole configs without a role_id, without
any secret_id source, or with an empty mount.
Also append the CredentialsUnavailable error variant used by the
fail-closed credential gate.
* feat(kms): implement AppRole login with background renewal and fail-closed expiry
Implement the AppRoleLogin token source (vaultrs approle login +
renew-self) and wire lease-bound credentials through the provider:
- Each successful login/renewal installs a new client generation in the
ArcSwap; in-flight requests finish on the generation they captured.
- A background renewal task refreshes at half the lease TTL: renewable
tokens are renewed in place, everything else (or a failed renewal)
falls back to a fresh login. Auth exchanges run under the typed retry
policy (OpClass::Auth) and failed cycles retry on a fixed cadence, so
the provider recovers once Vault does.
- Fail-closed: current() refuses to hand out a token inside the
configured safety window of its expiry (default: one attempt timeout),
returning CredentialsUnavailable instead of sending a request whose
token may lapse mid-flight.
- Refreshes are single-flight: concurrent triggers for the same
generation coalesce into one login.
- The renewal task's owner handle lives on the KMS service version:
stop() shuts it down explicitly and reconfigure recycles it via
cancel-on-drop when the old version is discarded.
- The secret_id file is re-read on every login attempt; missing or empty
files fail the attempt without contacting Vault. Crate-owned copies of
tokens and secret_ids are zeroized on drop, and Debug output of every
credential-carrying type stays redacted (leak regression tests).
The renewal machinery is covered by paused-clock tests driving a
scripted token source: renew-at-half-TTL timing, login fallback,
fail-closed window entry and recovery, prompt task recycling, and
coalesced concurrent refreshes.
* feat(kms): add Vault Agent token file authentication
Add the TokenFile source: the token is read from an agent-managed sink
file (RUSTFS_KMS_VAULT_TOKEN_FILE or the TokenFile auth config) and
re-read once per poll interval (default 30s) through the existing
renewal loop, so a token rotated by the agent installs a new client
generation within one poll of the atomic replace. Each successful read
extends the token's observed validity to twice the poll interval; a
file that disappears or turns empty keeps failing the refresh until the
fail-closed window trips, and heals the provider as soon as it is
restored.
Reads are strict and never contact Vault on failure: the file must be
non-empty after trimming, and on Unix group/other permission bits are a
hard error (mirroring the SFTP host-key rule). Rotation detection uses
a content digest; the token itself is never stored on the source and
the crate-owned copy is zeroized.
Configuring the token file together with AppRole or an explicit static
token is rejected as a configuration error. All new config fields are
serde(default) and the strict admin-configure deserializer accepts the
new variant.
Covered by paused-clock tests (atomic replacement installs a new
generation next cycle, deletion fails closed and recovers, prompt task
recycling) plus negatives for missing/empty/over-permissive files and a
Debug leak regression.
* docs(kms): add Vault authentication and credential lifecycle runbook
Cover choosing between static token, AppRole, and Vault Agent token
file auth; AppRole role setup with SecretID delivery and rotation;
Agent sink deployment with the permission requirements; and the
fail-closed window semantics with a troubleshooting table keyed on the
renewal task's log lines.
* feat(kms): retain historical master key versions for Vault KV2 rotation
Rotation previously had to be rejected outright because replacing the
stored material would orphan every DEK wrapped by earlier versions.
Vault KV2 now keeps each version's material in an immutable, create-only
record at {prefix}/{key_id}/versions/{N} and treats the top-level record
as the current-version pointer plus fast-path material copy:
- decrypt resolves the envelope's master_key_version to its version
record; a missing version fails closed with KeyVersionNotFound and
never falls back to the current material
- envelopes without a version (pre-versioning writers) resolve to the
baseline_version frozen at the key's first rotation, so never-rotated
keys behave exactly as before
- generate_data_key stamps the wrapping version from the same key record
snapshot that supplied the material
- rotate_key commits in check-and-set order: freeze baseline, persist
the next version's material, then switch the current pointer; any
failure leaves the current pointer untouched, and concurrent rotations
serialize on the CAS writes with monotonically unique versions
- key listings drop the versions/ directory entries and physical key
deletion purges version records before the key record
Refs rustfs/backlog#1565
* test(kms): pin rotation contracts across backends and document retention
Completes the backlog#1565 series with the cross-backend regression net
and operator documentation:
- Vault Transit: ignored integration test proving version-prefixed
historical ciphertext still decrypts after rotation, with no
RustFS-side version bookkeeping in the envelope
- Local: offline mixed-format test interleaving pre-versioning and
versioned envelopes through a rejected rotation; the existing
rotation-rejection pinning test already covers material immutability
- docs: kms-backend-security.md gains the KV2 versioned retention
model, version record retention/destruction preconditions, and the
upgrade-before-first-rotation cluster constraint
Refs rustfs/backlog#1565
Contract-only module for KMS backup/restore (no handler or backend
wiring): versioned manifest schema with completeness marker and sealed
digest, the (backend, at-rest protection) responsibility matrix, typed
fail-closed errors, and the zero-write restore dry-run report. Fields
whose shape depends on in-flight contracts are reserved and reject data
in format version 1.
Rotation previously had to be rejected outright because replacing the
stored material would orphan every DEK wrapped by earlier versions.
Vault KV2 now keeps each version's material in an immutable, create-only
record at {prefix}/{key_id}/versions/{N} and treats the top-level record
as the current-version pointer plus fast-path material copy:
- decrypt resolves the envelope's master_key_version to its version
record; a missing version fails closed with KeyVersionNotFound and
never falls back to the current material
- envelopes without a version (pre-versioning writers) resolve to the
baseline_version frozen at the key's first rotation, so never-rotated
keys behave exactly as before
- generate_data_key stamps the wrapping version from the same key record
snapshot that supplied the material
- rotate_key commits in check-and-set order: freeze baseline, persist
the next version's material, then switch the current pointer; any
failure leaves the current pointer untouched, and concurrent rotations
serialize on the CAS writes with monotonically unique versions
- key listings drop the versions/ directory entries and physical key
deletion purges version records before the key record
Refs rustfs/backlog#1565
* fix(kms): repair vault test call sites missed by the timeout refactor
Three offline tests still constructed VaultKmsClient with the pre-#5472
single-argument signature, leaving cargo test -p rustfs-kms unable to
compile. Pass the same 30s attempt timeout the neighbouring tests use.
* refactor(kms): route Vault clients through a rotatable credential provider
Both Vault backends previously built a VaultClient in their constructor
and held it for the lifetime of the backend, which leaves no seam for
re-authentication: rotating credentials would require tearing down the
whole backend.
Introduce backends/vault_credentials with a TokenSource trait (only
StaticToken for now; AppRole login and agent token files land in
follow-ups) and a VaultCredentialProvider that owns the authenticated
client behind an ArcSwap. Request paths take a per-call snapshot via
current(), so a future rotation swaps in a new client generation without
interrupting calls already in flight. Tokens held by this crate are
zeroized on drop, and Debug output of every credential-carrying type is
redacted (covered by a leak regression test).
Behavior is unchanged: static token, namespace, and per-attempt timeout
feed the same VaultClientSettings as before, and AppRole configurations
are still rejected at construction with the same message.
PR #5472 added a required attempt_timeout parameter to
VaultKmsClient::new, while PR #5474 (developed in parallel) added
three tests calling it with the old single-argument signature,
breaking compilation of cargo test -p rustfs-kms. Pass the same
30-second timeout the neighboring tests use.
PR #5472 added an attempt_timeout parameter to VaultKmsClient::new while
PR #5474 was developed in parallel and added three tests using the old
single-argument signature. Both merged cleanly at the text level, leaving
main unable to compile the rustfs-kms test target. Align the three call
sites with the current constructor signature.
* fix(kms): restore vault backend test compilation after timeout parameter
PR #5472 added an attempt_timeout parameter to VaultKmsClient::new while
PR #5474 landed tests still using the one-argument form, leaving
'cargo test -p rustfs-kms' unable to compile on main. Pass the same
30-second timeout the surrounding integration tests already use.
* feat(kms): add master key version to data key envelope contract
DataKeyEnvelope gains an optional master_key_version field recording
which KEK version wrapped the DEK, so rotation-aware backends can load
the matching historical material on decrypt. The field is skipped when
None, keeping envelopes from non-rotating backends byte-identical to
the historical seven-field JSON shape, and legacy envelopes without the
field deserialize to None. The envelope discriminator marker is
untouched, so mixed-format routing is unchanged in both directions.
Adds the KeyVersionNotFound typed error for version-addressed material
lookups that must fail closed instead of falling back to the current
version.
Refs rustfs/backlog#1565
