DeleteBucket answers from a raw per-disk residue scan rather than from a
listing, so it can refuse for a reason no S3 request can observe: the
client drains every version the API will show, DeleteBucket still returns
BucketNotEmpty, and the client-visible message is the generic "The bucket
you tried to delete is not empty" for every blocker kind.
The server does know which residue blocked it, and where — that is what
`bucket_delete_blocked` carries. But it was emitted at `debug`, below
both the `error` DEFAULT_LOG_LEVEL and the `info` the CI s3-tests lane
runs at, so it was never actually written down. An intermittent
BucketNotEmpty in that lane leaves a server log with no trace of the
refusal at all, which is not a diagnosable state: confirmed against the
artifact log of a failing run, where the rejected bucket appears only in
span-close lines and the blocker event is absent entirely.
Split the blocker kinds by whether the client can still reach the
residue. A visible version or a tier free-version is an ordinary 409 —
the bucket really is not empty and the caller can list and delete what is
left — so that stays at `warn`. UnknownXlMeta, OrphanDirectory, and
DiagnosticBudgetExceeded are on-disk state no S3 request can remove; that
is a server-side integrity problem and is now reported at `error`, with
the blocker kind, the residue counts, and the sample path.
This does not change what DeleteBucket accepts or rejects, and does not
retry or suppress anything — it makes the existing diagnosis reachable.
Refs #7005, #7010
fix(docs): update stale docs/README.md references to docs/architecture/README.md
The file docs/README.md was removed in a previous commit but references
in AGENTS.md, ARCHITECTURE.md, and CLAUDE.md were not updated. The
expanded check_doc_paths.sh now catches these stale references.
Update the workspace s3s pin and refresh the lockfile with cargo update/upgrade.
Remove the unused lifecycle url dependency reported by cargo shear.
Tighten the s3s footprint ratchet to the current observed baseline.
PR #7023 rewrote the handoff retry scripts with shell parameter
expansions collapsed into Actions expression syntax: ${GH_TOKEN:-},
${{attempt}}, ${{DISPATCHED:-0}}, ${{TITLE}} etc. GitHub parses
${{...}} as workflow expressions, and bare identifiers are invalid
there, so all seven shared-VM suite workflows (upgrade, s3-compat, kms,
tier, storage, heal, pool-expand) were rejected as invalid workflow
files on main.
Symptoms since 2026-09-01 23:11 +0800 (bba9347):
- every push to any branch produced 'failure' runs with no jobs
('This run likely failed because of a workflow file issue')
- the nightly functional chain dispatched rustfs-chain-upgrade at
17:08Z but the event was silently dropped: zero repository_dispatch
runs for all eight shared-VM suites overnight (only performance,
whose file was untouched, ran)
- the workflows API listed them by path instead of name
Fix: restore the shell expansions (${VAR}, ${VAR:-default}); quote the
expected-event name without legacy backticks; render the markdown fence
via printf so shellcheck can parse the block. actionlint and YAML
validation now pass clean on all eleven rustfs-*.yml workflows.
Relocate issue-triage and pr-review from the tool-specific .mimocode
directory into the shared .agents/skills tree that every agent already
reads (AGENTS.md, .claude/skills symlink), and ignore .mimocode/ so a
local copy never gets recommitted.
- New RustFS Replication Test workflow (rustfs-replication-test.yml):
standalone workflow_dispatch (suite selector bucket/site/all) and
repository_dispatch rustfs-chain-replication; runs on the shared
smoke-testing runner under the shared functional concurrency group.
- Suite never fails the workflow (continue-on-error): failures are filed
as redacted issues in rustfs/backlog (deduped per run) and the report is
uploaded to rustfs/dashboard functional-reports/replication/<date>.md.
- Security now hands off to Replication, making it the tenth and final
link: upgrade -> s3 -> kms -> tier -> storage -> heal -> pool ->
security -> replication (performance stays parallel on pf-testing).
- Depends on rustfs/auto-testing#27 (rustfs-replication-test.sh).
Co-authored-by: houseme <[email protected]>
Replication could fail an object with nothing in the server log an
operator could act on. Every failure branch in the resyncer is quieter
than `error` on purpose — most sit on the hot path and fire once per
object per ARN — but `DEFAULT_LOG_LEVEL` is `error`, so on a stock
deployment a failed object produced no line at all. Raising those
branches to `warn` (#6840) did not close this: the default filter still
dropped them.
Report the terminal outcome instead of the branches. `replicate_object_
with_outcome` and `replicate_delete_with_outcome` now emit one `error`
per failed (object, target) once the per-target results are merged,
carrying the object key, version id, target ARN and endpoint, and the
target's own error, redacted through `sanitize_resync_error_detail` so
an echoed credential cannot reach the log. Volume is bounded by objects
that actually fail rather than by attempts inside a transfer.
Also state the single-PutObject size limit instead of discovering it at
the target. Replication picks its transport from the source object's
storage shape, not its size, so an object written with one PutObject
replicates with one PutObject however large it is — and S3 caps that at
5 GiB. Such an object could never reach a generic S3 target, and only
found out after streaming the whole body. `replication_single_put_size_
error` fails it up front with a message naming the size, the limit, and
the remedy.
Version-identity drift moves to `error` on a 10-minute per-ARN throttle.
It was `warn` deduped once per ARN per process, so the one line
explaining why a purged version is still on the target was both filtered
out by default and gone for good after it first fired.
Fixes#6825
Refs #6822
`build_metrics_summary` emitted a single metric entry for the local
deployment with `online` hardcoded to `true` and `last_online` stamped
with the current time, so `mc admin replicate status` reported "I am
online" rather than whether the remote site was reachable. A peer could
be down for minutes with replication failing while the status page
stayed green, leaving operators with no signal that the link had
dropped.
Emit an entry for every peer instead, deriving `online` from the
`reachable_peers` set the handler already computes by probing each peer,
and take `total_downtime`/`last_online` from the replication heartbeat's
existing `EpHealth` tracking. Node-local replication counters stay on
the local entry so a two-site cluster does not double-count its own
traffic.
The new `BucketTargetSys::endpoint_health` accessor deliberately does not
call `init_hc`: unlike `is_offline` it must not create health entries as
a side effect, or merely rendering the status page would mark an unknown
peer online.
Failure counters (`Errors`) are unchanged and still read zero; that is a
separate defect in the bucket-level statistics path and is not addressed
here.
The repository_dispatch handoff step was continue-on-error with a single
attempt: if the call failed (token lacking contents:write, transient API
error), the chain stalled silently while every job stayed green.
Each handoff now retries 3x and, if all attempts fail, files an alert
issue in rustfs/backlog with the exact recovery command before exiting 1
(still continue-on-error, so suite workflows themselves never fail).
`local_idp_settings` stamped the site region into the reported OpenID
settings whenever the federated identity service was published, which it
is even with OpenID disabled and no provider configured. The add
preflight compares those settings verbatim, so two sites in different
regions could never be paired: `replicate add` failed with `IDP settings
mismatch` while both sites reported an identical, empty `identity_openid`
config.
Report the empty OpenID settings when no provider is configured, so the
region only qualifies real provider identities, and name the diverging
field in the rejection instead of emitting a bare mismatch. Scalar values
are echoed; nested objects and credential-derived leaves are reported by
presence only.
Fixes#7003
Problem: the nightly functional chain has not completed end-to-end.
Evidence from recent runs:
- workflow_run events are fire-and-forget: after KMS finished at 17:09Z
on 8/31 no tier run was created; rustfs-storage-test.yml has never run.
- 'if: conclusion == success' gates skip downstream suites on any
failure (security was skipped after pool failed on 9/1 01:48Z).
- rustfs-pool-expand-test.yml embedded a heal pass without
continue-on-error, so a heal failure failed the whole workflow.
Fixes:
- Add rustfs-functional-chain.yml: entry point that dispatches the first
suite via repository_dispatch; each suite hands off to the next with an
explicit, re-drivable API call instead of workflow_run triggers.
- Split heal out of the pool workflow (renamed to RustFS Pool Expansion
Test): heal now runs exactly once per chain, in rustfs-heal-test.yml
(storage -> heal -> pool).
- Every suite job gets continue-on-error so a failing test never fails
the workflow; failures are filed as issues in rustfs/backlog (report
+ redacted log tail) and the chain moves on.
- Clone rustfs/auto-testing with the PF token via 'gh repo clone' plus a
5-attempt retry loop (transient clone failures aborted whole suites).
- Stop rewriting functional/index.html from every suite (divergent
copies raced each other with stale SHAs); the canonical index now
lives in the dashboard repo.
- Standalone workflow_dispatch runs are unchanged and never forward the
chain; performance runs on its own runner, dispatched in parallel.
Six set_disk::ops tests failed non-deterministically only under
concurrent full-suite load, rotating between runs while each passed in
isolation. All six share one root cause: a lock-owning put_object
quorum-acks once the rename fanout reaches write quorum and lets a
detached tail task finish the lagging disks, so a fixture that inspects
per-disk state immediately after PUT can observe a disk the tail has not
reached yet.
The two heal report fixtures, the inline-commit fixture, and the
transaction-fencing fixture read or delete physical shards right after
PUT, and hit FileNotFound on a lagging disk. The two metadata-cache
fixtures prime the cache after PUT, and the read fanout refuses to publish
a cache entry while any disk still reports an error, so the priming read
observably published nothing.
Keep every affected setup PUT on the full-fanout commit path with
no_lock: true, following the existing precedent in this module, so PUT
returns only after every disk has committed. The option only governs lock
acquisition, so it does not weaken what any of these fixtures assert; the
transaction-fencing gate in particular is driven by the fleet proof and
env vars, never by the lock option. Where a fixture also depends on cache
publication, re-prime until the current generation is observably cached
instead of asserting on a single read that a loaded host can stall past
the cache TTL. The heal race fixture's shard damage injection is
best-effort by construction, so it now skips injection when the previous
round's tail still lags rather than unwrapping a read that may
legitimately race.
No production code changes, and no retries or sleeps added.
* test(ecstore): retain final decommission capacity snapshot override
take_decommission_capacity_info_override_for_test used to pop the queue
to exhaustion, after which get_decommission_all_pool_capacity_infos
silently fell back to the host's real statfs numbers. Any new sampling
point added to the decommission start paths re-introduced that host
dependency and broke tests on some dev machines (#6989 patched one
instance by topping up snapshot counts, but the coupling remained).
Keep the final queued snapshot and replay it for every subsequent
sample so tests always observe injected capacity once an override is
installed. All existing injection patterns (single snapshot, repeated
identical snapshots, decreasing sequences ending at the post-operation
state) keep their semantics.
* test(ci): serialize load-sensitive heal and cache-generation tests
Under a heavily parallel nextest run (~792 ecstore tests), two tests of
set_disk::ops::heal::heal_result_report_tests failed nondeterministically
per round (different members each time; all 29 pass standalone). Every
test in the module builds a TempDir-backed 4-disk hermetic erasure set
and drives MiB-scale writes plus deep-scan heal: under load a single
disk's IO can fail while write quorum still holds, flipping per-disk
readback and aggregate-outcome assertions. The module's #[serial]
markers do not serialize across nextest's process-per-test boundary.
Verification also caught complete_multipart_generation_retires_cached_snapshot
failing once under the same load; it and its object.rs sibling carry
#[serial(metadata_cache_invalidation_probe)] and assert
get_object_metadata_cache generation semantics - the same shape that
forced the transition matrix tests into the serial group.
Add both families to the ecstore-serial-flaky test-group in the default
and ci profiles. Preventive serialization only, no retries. Three full
parallel rounds after the change: 792/792 passed each round.
A completed multipart upload's staging cleanup prunes empty parent
directories up to the volume root, which removes shared prefixes such as
`data-movement/` and the per-object `<sha>/` while a concurrent
new_multipart_upload builds its destination chain below them. The writer
holds a descriptor to the pruned component, so its next handle-relative
mkdirat fails NotFound. Because rename never retried NotFound, the cleanup
fan-out failed several disks in the same window and broke write quorum.
Give rename preparation its own retry rule: a NotFound is retried once per
component below the base directory, so a rebuilt walk outlasts a pruning
walk, which removes ancestors monotonically upward and stops at the base.
A destination whose parent is the base keeps NotFound terminal, so
speculative cleanup renames still fail fast, and the base is only ever
opened, never created, so a genuinely missing base still fails. The rename
itself keeps its own budget and its unchanged NotFound-is-terminal rule.
scripts/check_test_wiring.py and scripts/check_security_coverage.py import
tomllib, which landed in Python 3.11. macOS ships /usr/bin/python3 at 3.9, so
`make pre-commit` failed on a clean machine with `ModuleNotFoundError: No
module named 'tomllib'` in test-wiring-check, even though the checkers
themselves are fine.
Add scripts/python_bin.sh, which resolves an interpreter (explicit
RUSTFS_PYTHON, then python3.14..3.11/python3/python on PATH, then a
`uv run --python 3.12 --no-project` fallback) and execs it, failing with the
concrete remediation when nothing usable exists. Route the Make call sites
through RUSTFS_PYTHON_BIN. CI workflows keep calling python3 directly because
their runners already provide 3.11+.
The mixed-version rolling upgrade suite asserted a single list_objects_v2
snapshot seconds after restarting a node. Peers keep a restarted node's
drive in Suspect/Returning for ~probe_interval(2s) x success_threshold(3),
and while one drive is excluded the strict listing quorum (write quorum,
3 of 4) drops objects that were themselves legally written at 3/4 during
an earlier node's identical post-restart window, under-counting the
listing (observed as 254 vs 258 in CI) even though every object still
GETs correctly. Replace the snapshot asserts with a bounded convergence
poll; a real upgrade data-loss regression still fails after the deadline.
* fix(release): normalize development package versions
* ci: build only the rustfs release binary
---------
Co-authored-by: Zhengchao An <[email protected]>
test(ecstore): assert inline fanout gate on deterministic scheduled metric
non_inline_data_read_early_stop_does_not_add_inline_fanout_on_unequal_layout
compared disk_call_counters::KIND_READ_VERSION totals between the two-phase
read-plan gate being off and on. That counter records inside each spawned
fanout task, so the single-pending inline hedge read races the early-stop
abort_all(): whether the hedge task gets its first poll before cancellation
decides a 4-vs-5 count per read. Under concurrent nextest load the two reads
can disagree (reproduced locally at ~5% when run beside one other test,
matching the CI failure on PR #6961).
Assert on the rustfs_io_get_object_metadata_fanout_scheduled histogram
instead, which records the scheduling decision synchronously in the fanout
loop and is deterministic, using the CapturingRecorder + current-thread
runtime pattern already used by the neighboring tests in this module.
Allow release builds to set RUSTFS_BUILD_VERSION at compile time while keeping the existing tag, short commit, and package-version fallback when the variable is unset or empty.
Co-authored-by: heihutu <[email protected]>
