The admin surface had accumulated one near-identical response helper per handler file. This folds the byte-equivalent ones into `rustfs/src/admin/utils.rs` so the wire shape of an admin JSON answer is pinned in one place instead of being re-derived twelve times.
Folded into `crate::admin::utils`:
- `json_response(status, &value)` — 9 local definitions removed: batch_job.rs, kms_backup.rs, oidc.rs, diagnostics.rs (identical signature), object_data_cache.rs and site_replication.rs (hard-coded `StatusCode::OK`, whose call sites now pass `StatusCode::OK` explicitly), ilm_transition.rs (arguments were `(&value, status)` and are swapped at every call site), and kms_key_metadata.rs / kms_key_lifecycle.rs (concrete response types now covered by the generic helper).
- `empty_response(status)` — 2 local definitions removed: site_replication.rs (`Body::empty()`) and table_catalog/mod.rs (`Body::default()`); `Body::empty()` is defined as `Body::default()`, so the two were already the same response.
- `extract_query_params(uri)` — 4 local definitions removed: kms_keys.rs (was `pub(super)`), replication.rs, batch_job.rs, config_admin.rs. All four bodies were behaviourally identical (`form_urlencoded::parse` over `uri.query()`, last-wins on repeated keys, valueless parameters kept as empty strings); they differed only in blank lines. kms_key_lifecycle.rs, which imported the kms_keys copy, now imports the shared one.
Intentionally left alone:
- heal.rs `json_response` — different shape: returns a bare `S3Response` (not `S3Result`) and additionally sets `CONTENT_LENGTH`.
- kms_rekey.rs `json_response` — same divergent shape as heal.rs: bare `S3Response` over already-serialized `Vec<u8>`.
- idp_compat.rs `json_response` — encrypts the payload via `encode_compatible_admin_payload`; it is not a duplicate of the plain JSON helper.
- scanner.rs `json_response` — takes raw `Vec<u8>`, and `ScannerCycleStateResetHandler` genuinely passes a byte literal rather than a serializable value, so the local helper stays.
- oidc.rs `extract_query_param` — singular, returns `Option<String>` for one key, hand-rolls its own splitting via the `urlencoding` crate; a different function, not a variant of the map builder.
Wire behaviour on the success path is byte-identical everywhere: same status, same `Content-Type: application/json` (every local copy spelled the same value, whether via a per-file `JSON_CONTENT_TYPE`/`CONTENT_TYPE_JSON` constant, `HeaderValue::from_static`, or `"application/json".parse()`), same serialized body bytes, and no other header. The only behavioural change is the message text on the serde-serialization-failure arm, which is now uniformly `failed to serialize response: {e}`; that arm is unreachable for these owned response structs and the acceptance criteria pin only status and content type.
No `include_str!` self-grep assertion needed updating: the affected tests in ilm_transition.rs, site_replication.rs, kms_keys.rs, kms_key_metadata.rs, kms_key_lifecycle.rs, object_data_cache.rs, and table_catalog/tests.rs are all bounded by handler `impl Operation` / entry-point markers that sit well after the removed helpers, and none of them assert on a `json_response`, `empty_response`, or `extract_query_params` string.
Tests: `rustfs/src/admin/utils.rs` gains `json_response_carries_status_content_type_and_serialized_body`, `json_response_reports_serialization_failure_as_internal_error`, `empty_response_has_no_body_and_no_headers`, `extract_query_params_decodes_percent_escapes`, and `extract_query_params_keeps_valueless_parameters_and_survives_no_query`. The percent-decoding coverage previously in batch_job's `extract_query_params_decodes_job_id` moves there, and batch_job keeps its own end-to-end coverage as `require_job_id_decodes_and_rejects_missing_and_empty`.
Reference: rustfs/backlog#1829 T6
Remove unused direct dependency declarations found by cargo-shear and delete the unlinked ecstore mimalloc diagnostics file.
Keep feature-forwarding dependencies explicit with package-local cargo-shear ignores so hotpath feature propagation remains intact.
Co-authored-by: heihutu <[email protected]>
* refactor(ecstore): drop the client shim, import rustfs-s3-client directly
Completes the migration window opened by the rustfs-s3-client extraction (rustfs/backlog#1842 PR3): every consumer now imports the client crate directly and the crate::client shim is deleted.
- All in-crate crate::client:: paths (tier warm backends, tier core, lifecycle tier_sweeper, replication storage boundary, set_disk) now import rustfs_s3_client::* directly; crates/ecstore/src/client/mod.rs and the lib.rs mod client declaration are gone.
- The two server-side modules historically misfiled under client/ move to their real homes: object_api_utils.rs to crates/ecstore/src/object_api/ (it builds engine-side object readers/writers), and object_handlers_common.rs to crates/ecstore/src/bucket/lifecycle/ (it is the lifecycle noncurrent-version cleanup helper). The latter now routes its replication calls through the lifecycle replication_sink boundary (schedule_delete wrapper and the sink's ReplicationObjectBridge re-export), as the lifecycle guard requires.
- The ecstore public facade drops api::client: object_api_utils is exposed as api::object_api_utils, and the rustfs crate takes admin_handler_utils (AdminError) from rustfs-s3-client directly (new dependency).
- Guard updates: the migration guard no longer pins mod client in ecstore's lib.rs or the admin_handler_utils facade module (it pins the new api::object_api_utils facade instead), and the module-lint register follows object_api_utils.rs to its new path.
Verification: cargo check -p rustfs-ecstore --all-targets and -p rustfs; cargo fmt --all; tier/transition/lifecycle-focused nextest (626 passed) and the decommission/rebalance/heal families in a filtered run (603 passed; the full-suite parallel run only fails on this machine's known decommission/rebalance baseline flakes, which pass in filtered reruns and fail identically on pristine origin/main); layer/migration/s3s/logging/error-format/doc-path guard scripts all pass.
* docs(architecture): record the S3 client extraction and reword invariant 4 (#6669)
Closes the documentation step of rustfs/backlog#1842. ARCHITECTURE.md invariant 4 now states the serving-vs-consuming distinction the adversarial ruling asked for: ecstore must not serve HTTP/S3 wire types, while consuming remote S3 endpoints is a legitimate engine capability that lives in the extracted rustfs-s3-client crate. The violation note is updated from the pre-extraction snapshot (58 files, embedded client) to the current ratcheted state (shrink-only S3S_ECSTORE_FILES_BASELINE in scripts/check_s3s_footprint.sh, object_lock converted first), and the crate map gains s3-client. ecstore-module-split-plan.md gets the client-directory entry the plan was missing: a Current Shape row and a completed-extraction section describing the pure-move + shim + direct-import sequence and the re-homing of the two misfiled server-side modules.
* refactor(rustfs): carve app/object out of object_usecase.rs — shared, extract, test_support children (backlog#1841 step 1)
Mechanical move-only split of rustfs/src/app/object_usecase.rs (19.7K lines). The file body moves to rustfs/src/app/object/mod.rs, and the first self-contained slices move into children: shared.rs (cross-cutting helpers: quota admission, response checksum injection, object-lock write validation, table-catalog mutation guard, deadlock request guard, proxy passthrough utilities), extract.rs (snowball auto-extract path incl. tar/pax helpers and execute_put_object_extract), and cfg(test) test_support.rs for cross-module test scaffolding. object_usecase.rs stays as a thin pub use facade so every existing crate::app::object_usecase:: path keeps working.
No behavior change: items move verbatim; the only source edits are visibility widenings required by the new module boundaries (private -> pub(super); pub(super) -> pub(crate) for the three helpers multipart_usecase and the app gating tests import). Guard scripts that pinned rustfs/src/app/object_usecase.rs now scan the rustfs/src/app/object tree, and the table_catalog source-text guard test concatenates the split files.
* refactor(rustfs): move the GetObject read path into app/object/get.rs (backlog#1841 step 2)
Move-only continuation of the object_usecase split: cold-fill orchestration, disk-permit admission, streaming readers and resume control, stream-buffer tuning, execute_get_object / execute_get_object_attributes, the GET replication proxy helpers, and their unit tests move from app/object/mod.rs into app/object/get.rs. Items keep their original text; cross-module call sites rely on the visibility widenings introduced in step 1.
* refactor(rustfs): move the PutObject and CopyObject paths into app/object (backlog#1841 step 3)
Move-only continuation: put.rs takes the PUT body admission and timeout readers, zero-copy and eager-commit machinery, execute_put_object, and the PUT unit tests; copy.rs takes the copy namespace/lifecycle lock helpers and execute_copy_object with its tests. Two source edits beyond visibility widenings: PutObjectChecksums fields become pub(super) (read by shared::apply_trailing_checksums across the new module boundary) and one relative super::storage_api call in the copy path becomes crate::app::storage_api since super now resolves to app::object. The table_catalog source-text guard concatenates the new files.
* refactor(rustfs): finish the object_usecase split — delete, head, restore modules (backlog#1841 step 4)
Move-only completion: delete.rs takes the delete helpers, cfg(test) delete hooks, and execute_delete_object/execute_delete_objects; head.rs takes execute_head_object with the HEAD replication proxy helpers; restore.rs takes execute_restore_object. app/object/mod.rs is now just the shared import prelude, module wiring, and the DefaultObjectUsecase struct with its constructors, accessors, and the execute_select_object_content delegation; the emptied tests module is gone. The delete re-export glob is cfg(test)-gated because its only cross-module consumers are the delete test hooks.
The table_catalog source-text guard now isolates the delete entrypoints from app/object/delete.rs, and doc/comment references that pointed at rustfs/src/app/object_usecase.rs internals now point at the per-operation modules.
The object_lock module evaluated WORM state through s3s wire DTOs (ObjectLockRetention, ObjectLockLegalHold, DefaultRetention, Date) and s3s header constants, keeping the storage engine coupled to the serving protocol (rustfs/backlog#1842, ARCHITECTURE.md invariant 4). This PR gives the module its own storage-level vocabulary and pushes the DTO conversions to the boundaries that already speak s3s.
New crates/ecstore/src/bucket/object_lock/types.rs defines RetentionMode, LegalHoldStatus, ObjectRetention, ObjectLegalHold, and DefaultRetention with no s3s dependency. objectlock.rs parses persisted metadata into these types using the rustfs-utils lowercase header constants (the same literal keys as before, pinned by the existing g-key-002 test). objectlock_sys.rs evaluates retention/legal-hold/default-retention from them; the fail-closed error messages and decision logic are unchanged line for line where possible.
Boundary conversions:
- bucket/metadata_sys.rs gains default_retention_from_object_lock_config, converting the persisted s3s configuration into the storage-level DefaultRetention; a rule without a usable GOVERNANCE/COMPLIANCE mode converts to None exactly like the evaluation code always ignored it, and days/years pass through so an invalid period still fails closed at evaluation time.
- check_object_lock_for_deletion_with_config becomes check_object_lock_for_deletion_with_default_retention (it only ever read the default retention); the lifecycle object_lock_boundary keeps the old s3s-typed signature and converts.
- The ObjectLockApi / ObjectLockStatusExt trait impls for the s3s DTOs move next to the persisted configuration owner in bucket/metadata.rs; the traits stay in object_lock/mod.rs.
- check_retention_for_modification now takes Option<RetentionMode>. The serving-layer wrappers (rustfs storage_api, set_disk options path) convert the request string with the new RetentionMode::parse_exact, which accepts only the canonical spelling — preserving the historical literal comparison where a non-canonical requested mode reads as a mode change and stays blocked.
- rustfs app-layer wrappers return the storage types; the replication-overwrite gate in object_usecase.rs uses the typed API (legal_hold.is_on(), RetentionMode::Compliance).
Ratchet: the ecstore-scoped s3s counter drops 42 -> 39 and the repo-wide file counter 211 -> 208 in scripts/check_s3s_footprint.sh.
Verification: cargo check -p rustfs-ecstore --all-targets and -p rustfs (lib+bins); cargo clippy -p rustfs-ecstore --all-targets and -p rustfs --lib --bins (clean); cargo nextest run -p rustfs-ecstore --no-fail-fast (4534/4542; the 8 failures are the same store::rebalance / store::heal machine-baseline set that fails identically on pristine origin/main, plus one fencing flake that passes in isolation); all object_lock/retention/legal-hold tests pass; guard scripts (layer deps, migration rules, s3s footprint, logging, error-format ratchet, doc paths) pass.
Upgrade rustfs-mimalloc and rustfs-mimalloc-sys to 0.5.1, then call the new safe wrapper from Tokio worker thread startup so mimalloc can treat runtime threads as threadpool workers.
Keep the hint no-op on Windows, matching RustFS allocator platform boundaries.
Co-authored-by: heihutu <[email protected]>
The INVENTORY_UID constant is only referenced inside
#[cfg(target_os = "linux")] test functions, so it appears unused on
macOS. Add a cfg_attr to allow dead_code on non-linux targets.
refactor(protos): move internode compat manifest send-site assertions into owning crates
Promotes the rolling-upgrade dual-write manifest from a test-only constant in rustfs-protos into the public rustfs_protos::compat_manifest module, moves the JSON-encoder send-site assertions into the crates that own the asserted sources (ecstore remote_disk.rs for requests, the rustfs binary node_service/disk.rs for responses), and splits the scanner Phase-0 overlap inventory so its heal- and ecstore-owned halves live in those crates. Adds a cross-crate include_str!/include! guard with fixture self-tests to scripts/check_layer_dependencies.sh so a library crate can never again read another crate's Rust source at compile time, and records the rule in docs/architecture/crate-boundaries.md.
Part of rustfs/backlog#1884.
The dynamic-configuration flow persisted KmsConfig to cluster storage as
raw JSON, leaving inline authentication material - the Vault token, an
AppRole secret_id, the Local master key - in config/kms_config.json in
cleartext.
Add rustfs_kms::config_secret: with the per-node RUSTFS_KMS_CONFIG_SECRET
set, those field values are sealed in place before persistence (Argon2id
with the Local key store's parameters + AES-256-GCM, per-value random
salt, the field's logical label bound as AEAD associated data so sealed
values cannot be swapped between fields). Sealed values carry the
versioned prefix RUSTFS-KMS-ENC[v1]:.
Compatibility is warn-only by owner decision: an unset secret keeps the
plaintext format and warns naming the exposed fields; plaintext values
load forever and reseal on the next save. Sealed values fail closed on a
missing or wrong secret. The sealing secret must be an independent trust
root - reusing the Local master key or Static secret is refused,
mirroring the backup-KEK rule.
perf(sse): classify GET response headers without a second KMS unwrap
An SSE-KMS GET performed two backend Decrypt calls per request: the
object layer's encryption resolver unwraps the envelope to build the
decrypted stream, and the S3 layer then called sse_decryption again
purely to derive response headers, discarding the returned key bytes.
Replace the S3-layer call with classify_sse_read_response, which
reproduces that call's behavior from stored metadata alone: SSE-C
validation errors and precedence, per-key kms:Decrypt authorization
ahead of every other failure mode, and the request's KMS audit summary
fields. The success outcome stays honest because a failed unwrap aborts
the read in the object layer before response classification is reached.
Tests cover header parity against the unwrap-based path, audit-tag
parity for allowed and denied principals, SSE-C validation parity, and
prove classification needs no DEK provider at all.
* feat(madmin): add account and two-factor wire contract
Defines the self-service account and MFA API shapes in one place so the
console and the `rc` CLI decode identical payloads instead of each
carrying its own copy of the contract.
`AccountMutability` is part of the contract on purpose: a client needs to
know whether the server will accept a password change for this identity
before offering the control, rather than discovering it from a rejected
request.
* feat(s3-types): add IAM identity audit events
Adds `iam:Identity:CredentialChanged` and `iam:Identity:AuthChallenge`
so account and authentication activity reaches the audit pipeline in its
own namespace, the way the KMS events already do. Neither is reachable
from a bucket notification config.
Two variants for the whole surface rather than one per operation:
`mask()` gives every variant its own bit in a `u64`, and the budget is
nearly spent (63 of 64 used after this). The per-operation detail lives
in `AuditEntry::api.name` and the `iamOperation` tag, which is what a
SIEM filters on anyway. Splitting these further needs `mask()` widened
first.
* feat(iam): add two-factor authentication primitives
Implements the state machine behind TOTP enrollment and verification in
the IAM domain, so the admin handlers stay HTTP plumbing and the console
and CLI drive identical logic.
* `totp`: RFC 6238 over the workspace's existing hmac/sha1, pinned to the
published Appendix B vectors. SHA-1, 6 digits, 30s: the parameters every
mainstream authenticator app implements. Verification returns the
matched time step so the caller can burn it.
* `recovery`: ten single-use codes, 100 bits each, in a Crockford base32
alphabet without I/L/O/U. Stored as domain-separated SHA-256 digests —
a password KDF would have to run once per stored code on every attempt,
turning each guess into an attacker-controlled cost, and with uniform
100-bit input there is no dictionary for it to defend against.
* `challenge`: stateless HMAC tokens. A TTL cache would be node-local, so
a cluster without session affinity would issue on one node and verify
on another; nothing here needs replicating.
* `record`: two-phase enrollment, replay high-water mark, and lockout.
Pending enrollment never gates a login, so a mis-scanned QR cannot lock
an operator out, and re-configuring keeps the old factor working until
the new one is confirmed.
* `store`: one object per identity under `config/mfa/`, a sibling of
`config/iam/` so the IAM cache loader's startup walk does not sweep it
up. Optimistic `If-Match` writes; deliberately uncached, because a cache
would need cluster-wide invalidation to keep the replay mark and the
lockout counter honest.
* `qr`: server-side rendering, so neither client needs a QR encoder.
Enrollment is refused without `RUSTFS_IAM_MASTER_KEY`. A TOTP secret is
credential-equivalent, and one written in plaintext could be lifted off a
disk — worse than no second factor, because the user believes they have
one. IAM identities tolerate a missing master key for backward
compatibility; a new feature has no such history to honour.
Also adds `IamSys::revoke_sts_sessions_for_parent`, so a credential
rotation can invalidate the sessions minted under the old secret.
* feat(admin): add self-service account endpoints and the two-factor login gate
Adds the account surface (`/v3/account/*`), the second-factor endpoints,
the administrative reset (`/v3/user/mfa`), and `PUT
/v3/set-user-secret-key`, plus the gate on `AssumeRole`.
What the gate covers, and what it deliberately does not:
* `AssumeRole` is the only interactive login RustFS has, so it is where a
second factor can be enforced. With one enrolled it requires
`TokenCode`; without an enrollment the code path is unchanged, so
existing deployments are untouched.
* A request signed directly with a long-term access key stays ungated.
Gating it would break every script and CLI the moment a human enabled
2FA on their own account, and would add no protection: whoever holds
the secret key already has full access without presenting a code. This
is the division AWS draws; making 2FA meaningful for API access needs an
`aws:MultiFactorAuthPresent` policy condition, tracked separately.
`SerialNumber`/`TokenCode` are STS's own parameters, so an SDK or script
authenticates the same way the console does.
`caller_identity` resolves who a request acts as. The console signs with
a short-lived STS session, so "the caller" is almost never the key that
signed. It reports two separate capabilities: root cannot rotate its
secret (a process-wide `OnceLock` that also derives the internode RPC
secret) but *can* enroll a second factor — conflating the two would leave
the default deployment's console login unprotectable.
The self-service routes carry no admin action. Giving them one would be
wrong in both directions: it would stop an ordinary user from changing
their own password, and let any holder of that action change someone
else's. They gate on possession of the credential plus, for the
mutations, knowledge of the current secret — a signature only proves a
credential was used, so without that a hijacked tab could rewrite the
account's credentials or strip its second factor.
`set-user-secret-key` exists because the only prior way to change a
password was to re-POST the whole user through `add-user`, which rewrote
`status` and dropped the policy field — a password reset that silently
re-enabled a disabled account.
Wrong, replayed and malformed codes are indistinguishable on the wire;
the distinction survives only in the audit trail, where no submitted
value, secret or code is ever recorded.
* test(e2e): cover the two-factor lifecycle and its regressions
Unit tests cover the state machine at its edges; only an end-to-end test
proves the pieces are wired together and that the existing
authentication paths still behave.
Asserts, against a real server: enrollment is refused without a master
key; the full enroll/activate flow works with a genuine RFC 6238 code;
`AssumeRole` refuses without a factor and accepts a valid one; a recovery
code works exactly once; a direct SigV4 admin request keeps working with
a factor enrolled; `AssumeRole` for an unenrolled identity is unchanged;
and a password rotation invalidates the old secret.
The test computes TOTP codes itself rather than calling the server's
implementation — a shared helper could agree with a bug on both sides.
This suite caught a real defect during development: enrollment was
refused for root because its *password* is immutable, which would have
left the default deployment — an administrator signing into the console
as root — unable to protect the one login the feature exists for.
* docs(operations): document the two-factor authentication model
Records what the second factor protects and what it deliberately does
not, because several of the boundaries look like gaps until the
alternative is spelled out: why direct SigV4 access stays ungated, why
root credentials cannot be rotated at runtime, why secret keys cannot be
hashed in an S3 server, and why at-rest protection is mandatory for a
TOTP secret but optional for an IAM identity.
Also states the limitations plainly, including that GHSA-m77q-r63m-pj89
is unaffected: a holder of the root secret can still forge a session
token, 2FA claim included.
Placed alongside the other authentication and KMS security documents
rather than under a new `docs/security/`, which `.gitignore` excludes.
* fix(admin): route the new account handlers through the admin s3 facade
Two of the guardrails in the CI "Quick Checks" job rejected the previous
commits, so the required check would have gone red as soon as a maintainer
approved the workflow run.
`check_architecture_migration_rules.sh` requires everything under
`rustfs/src/admin` to reach `ECStore` through a domain module rather than
the root of `storage_api`. The MFA handler and the two `AssumeRole`
signatures now use `storage_api::runtime::ECStore`, which is where the
other ten admin handlers already take it from.
`check_s3s_footprint.sh` ratchets two counters that new code may not grow:
files referencing `s3s` and error-macro invocation lines. This branch added
four files and thirty-two lines to them. The ratchet is lower-only and its
header forbids raising a baseline to get green, so the construction moves
behind the facade instead: `storage_api::s3` now re-exports the request and
body types these handlers need and gains an `error` constructor over
`S3Error::with_message`. That is the same constructor the macro expands to
and the one `handlers/mod.rs`, `rebalance_internal_error` and
`invalid_object_lock_configuration` already call, so this is the existing
practice rather than a new one, and it keeps the `s3s` dependency in the
boundary file the s3gate migration replaces.
Every error code and message is carried over unchanged. In `sts.rs` only
the call site this branch added is converted; the sixteen that predate it
are left alone, because rewriting them would put unrelated churn in a
feature PR and push the counter below the baseline it is meant to hold.
Run replication resync target reconcile and follow-up resync recovery in a background startup task so bucket metadata transaction lock contention cannot keep a node from joining the cluster.
Co-authored-by: heihutu <[email protected]>
* fix(ecstore): handle metadata-less bucket residue
Diagnose metadata-less on-disk residue before non-force DeleteBucket reaches physical deletion, and keep scanner-discovered metadata-missing objects on a non-destructive heal path.
Add explicit heal --remove cleanup for unversioned metadata-less data directories, using the existing data-dir delete primitive and fail-closed shape checks so pre-commit or unknown residue is preserved.
Co-Authored-By: heihutu <[email protected]>
* fix(connect): adapt offline array validator
Wrap the filesystem summary validator in a closure so Option::is_some_and can pass the concrete array reference accepted by serde_json::Value::as_array.
Co-Authored-By: heihutu <[email protected]>
* fix(connect): remove redundant offline test clones
Move the temporary path into the swap closure after deriving the output path, keeping clippy's redundant-clone lint clean for offline bundle tests.
Co-Authored-By: heihutu <[email protected]>
---------
Co-authored-by: heihutu <[email protected]>
