* refactor(replication): split four oversized hot-path functions into focused helpers
Pure-move decomposition of the four oversized functions flagged by the
replication compatibility review (P1-18), unblocking migration milestone
M2 which requires resyncer moves to stay mechanical:
- resync_bucket (522 lines -> 61-line step sequence): leader lock,
target resolution, walk/collector/worker spawning, and dispatch loop
extracted into focused helpers; pure decision helpers (DTO builders,
HEAD-result classification) separated from IO orchestration.
- replicate_all (411 lines -> 113-line main body): initial target-info
seeding, read/stat option builders, skip-path notes, target HEAD
action resolution, and the multipart/single-put payload transport
extracted as private free functions.
- start_mrf_processor (306 lines -> 46-line spawn body): recovery guard,
ledger load, per-entry replay (delete/object/metadata), and retained
entry resolution extracted; retry bookkeeping semantics preserved
exactly (inner continue-paths push inside helpers, outer Missed push
stays in the loop).
- apply_iam_item (255 lines -> match dispatch skeleton): one helper per
IAM item type.
No behavior change: log texts, error paths, event emissions, and metric
counts are byte-identical; existing tests unchanged and green (238
ecstore replication/mrf/resync + 232 rustfs site-replication).
* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172)
* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets
Implements the MinIO active-active read-proxy protocol (P1-5 of the
replication compatibility review): when a GET/HEAD/GetObjectTagging/
PutObjectTagging/DeleteObjectTagging request fails locally with
not-found and the bucket has replication targets, the request is proxied
to the targets in rule order, mirroring bucket-replication.go
proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget.
Protocol surface:
- Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed
into ObjectOptions (proxy_request + proxy_header_set, matching MinIO
ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY
value is never re-proxied. Outbound client proxy calls send the marker
as "true"; replication worker convergence HEADs send it as "false" so
a peer's proxy layer cannot answer a convergence check by proxying
back to the source (which would fake Completed without a PUT).
- Target selection: new replication_proxy.rs get_proxy_targets — empty
when the marker is set, versioning is suspended, or no replication
config; otherwise filter_target_arns -> TargetClient lookup, skipping
targets with proxying disabled.
- TargetClient gains head_object_for_proxy/get_object (streaming) and
the three tagging calls. Proxy calls never send the replication-check
SSE-C exemption header; customer SSE-C keys are forwarded verbatim so
the target performs real decryption. Conditional (If-*) headers are
not forwarded (MinIO parity); Range and part_number are, with
parts_count/tag_count/storage_class/expiration passed through.
- Metrics: proxy counters now count only real client proxy traffic,
MinIO-aligned (one total per proxied request, one failed when no
target served it). The previous misattributed counters — replication
worker HEAD/PUT (#2672) and local tagging operations (#2682) — are
removed; ReplProxyMetric now maps the tagging counters instead of
dropping them.
e2e (fake_s3_target extended with tagging + header journaling): proxied
GET body + outbound header contract (marker present, no
replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero
outbound requests, GetObjectTagging, and metric mapping unit tests.
Rolling note: proxying only activates for buckets with replication
targets; requests carrying the marker keep pre-upgrade behavior.
Refs rustfs/backlog#1675 (P1-5)
* fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178)
SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport
headers. A MinIO/generic-S3 target silently discards them, storing bare
ciphertext with no decryption material — yet the PUT succeeded, so the object
reported COMPLETED with a silently unreadable replica (backlog#1675 N2).
Fail-closed design:
- SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in
BucketTargetSys per target ARN with a recording timestamp. Entries reset
whenever the target is rebuilt, edited, or removed (arn_remotes_map
lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes):
an expired verdict in either direction is re-earned through the audit, so
an Unsupported target recovers automatically after an upgrade (at most one
wasted PUT+HEAD audit per bad target per TTL window) and a Supported
verdict cannot outlive a backend swapped behind the same endpoint.
- Replication worker (replicate_object and replicate_all): fresh Unsupported
targets never receive the PUT — the attempt fails immediately into the
normal MRF retry channel with a "run ?replication-check to re-probe" hint.
Unknown or expired verdicts are audited: after the PUT the worker HEADs
the replica back through the replication-check channel (source version id
mapped through resolve_read_api_version_id, so null-version objects audit
correctly) and requires SSE-C evidence (the echoed customer-algorithm
header); missing evidence records Unsupported and fails the attempt.
Convergence HEADs are audited the same way, so a broken ciphertext replica
from an earlier attempt can never launder itself into COMPLETED via an
ETag match. The gate/evidence policy is pure (replication_target_boundary,
staleness folded in as an input) for the M2 worker migration.
- replication-check grows an SsecPassthrough probe phase: a probe PUT
carrying the live transport-header shape, HEAD-back for evidence, and a
machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure.
The probe verdict is synced into the runtime capability cache. Unlike
VersionFidelity, a failed SsecPassthrough phase does NOT fail the target
overall — it is a capability limit, not a broken replication contract,
and a plaintext-only deployment against such a target must not turn red.
- fake_s3_target: default mode now models a RustFS target (stores the
transport headers, echoes SSE-C evidence); the new
drop_unlisted_replication_headers mode models MinIO. The journal records
whether a request carried transport headers.
Receiver-echo verification: the replication-check HEAD exemption only skips
SSE-C key validation; the response has always built sse-customer-algorithm
from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver
change was needed — pinned end to end by the replication-check e2e against
a real RustFS target.
Rolling-upgrade constraint: RustFS targets older than the replication-check
HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail
it outright), so SSE-C replication to such targets reports FAILED. This is
deliberate — FAILED-and-retryable beats a silently undecryptable replica —
and self-heals: once the target is upgraded, the next TTL expiry (or a
manual ?replication-check re-probe) re-audits and records Supported.
Plaintext and managed-SSE replication are unaffected. The capability cache
is per-node; each node audits independently.
Known limitations:
- The audit judges evidence from the echoed customer-algorithm header only.
A hypothetical target that preserves that one header while dropping other
transport headers (partial-drop) would pass the audit; no known target
behaves this way — observed targets drop the whole unknown-header family.
- A mixed-version target cluster can flap the verdict between audits routed
to different target nodes until the rollout completes; the TTL bounds how
long each stale verdict persists.
New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a
header-dropping fake (FAILED + no second PUT via the capability cache,
journal-asserted; red run showed the old COMPLETED), replication-check
reports the SsecPassthrough phase Code while the target stays OK overall,
SSE-C heal convergence after a real target outage, and SSE-C
existing-object resync landing a REPLICA readable with the customer key.
TTL expiry in both directions is pinned at the cache and gate seams.
* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) (#6180)
* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2)
Pure-move milestone M2 of the ECStore replication split (backlog#1675
P1-17): relocate the resyncer's IO-free decision helpers, with their unit
tests, into the crates they already belong to by type ownership. No
behavior change.
Moved into crates/replication:
- resync.rs: resync_status_duration
- delete.rs: resync_existing_delete_replication_info,
replicate_delete_outcome, target_delete_version_id,
delete_marker_purge_version_id, delete_marker_purge_mrf_entry
- object.rs: version_identity_drifted, is_replication_target_offline_error,
SsecPassthroughCapability, SsecPassthroughGate, ssec_passthrough_gate,
ssec_passthrough_evidence_present (param-demoted to the echoed
customer-algorithm string; ECStore keeps the HeadObjectOutput adapter)
- filemeta.rs: NULL_VERSION_ID wire literal (crate-owned copy per the
filemeta-independence contract)
ECStore rewiring (Rule #14: imports stay in *_boundary.rs):
- resync/object-decision/target boundaries re-export the moved symbols;
resyncer call sites are unchanged
- bucket_target_sys keeps only the verdict cache + TTL and re-exports the
capability enum so existing consumer paths keep compiling
Not moved (signatures carry ECStore or aws-sdk types):
verify_resync_head_result, resync_target_error_detail, the SdkError
classifiers, the replicate_all_* option/info builders, and the env-coupled
bounded_resync_max_jobs admission clamp. README milestone table updated.
* chore(replication): retire the datatypes.rs relay early
README sanctions retiring datatypes.rs ahead of M4. The module was a
pure relay (resync boundary -> datatypes -> mod.rs facade) with no
external consumer importing it directly, so the facade now re-exports
ResyncStatusType from replication_resync_boundary and the relay file is
deleted. Consumers stay behind the ECStore facade, keeping Migration
Rule #15 intact — the original retirement wording ("consumers import
through rustfs-replication directly") conflicted with that rule and is
corrected in the README.
* chore(arch): extend migration guards to the M2-moved decision contracts
The adversarial review of the M2 move found the per-symbol ratchet in
check_architecture_migration_rules.sh was not extended for the moved
symbols, leaving them free to be redefined in ECStore or imported past
their boundary without CI noticing:
- resync definition pin + boundary fences gain resync_status_duration;
- the object-decision boundary fences gain the five delete-family
helpers (delete_marker_purge_mrf_entry, delete_marker_purge_version_id,
replicate_delete_outcome, resync_existing_delete_replication_info,
target_delete_version_id);
- the target-boundary fence gains the SSE-C gate family, the offline
classifier, and version_identity_drifted;
- a new definition pin rejects ECStore redefinitions of the M2-moved
fns/enums (ssec_passthrough_evidence_present deliberately excluded:
ECStore keeps a thin HeadObjectOutput adapter under that name).
Mutation-verified: a probe fn ssec_passthrough_gate under
crates/ecstore/src/bucket/replication trips the new pin.
Also anchors the intentionally-duplicated NULL_VERSION_ID wire literal
from the filemeta side and tightens the M2 README note on
bounded_resync_max_jobs.
Authoritative inventory of everything under scripts/ (backlog#1153
infra-13). One row per top-level entry; subdirectories get one row each and
keep their own READMEs. The test-layer map that ties the major runners
together lives in docs/testing/README.md.
Statuses
ci-gate — wired into CI, release, or image-build pipelines. Do not move,
rename, or change flags without updating the wiring listed in the last
column.
dev-tool — run by humans: local dev loops, runbooks, validation
harnesses. Kept working, not wired into CI.
archived — one-shot scripts whose investigation/issue is finished, moved
to scripts/archive/. Unmaintained reference material: never
wire into CI, and expect bit-rot. To resurrect one, move it back and give it
an index row here.
Adding a script? Add an index row in the same PR. Issue-scoped scripts
(run_issueNNN_*, validate_issue_NNN_*) are expected to be archived when
their issue closes.
Repository & CI gates
Entry
Status
Purpose
Wiring / docs
check_architecture_migration_rules.sh
ci-gate
Architecture-boundary anti-regression guard
ci.yml Quick Checks; make pre-commit
check_body_cache_whitelist.sh
ci-gate
Keeps the app-layer body-cache eligibility gate fail-closed
ci.yml Quick Checks
check_doc_paths.sh
ci-gate
Fails when instruction/architecture docs reference repo paths that no longer exist
make pre-commit / pre-pr
check_extension_schema_boundaries.sh
ci-gate
Extension-schema crate boundary guard
ci.yml Quick Checks; make pre-commit
check_layer_dependencies.sh
ci-gate
Crate-layering DAG guard (reads layer-dependency-baseline.txt)
ci.yml Quick Checks
check_logging_guardrails.sh
ci-gate
Blocks legacy logging patterns from returning
make pre-commit / pre-pr
check_migration_gate_count.sh
ci-gate
Migration-critical test gate with committed count floor (.config/migration-gate-floor.txt)
ci.yml Test and Lint; docs/testing/README.md
check_no_planning_docs.sh
ci-gate
Blocks committed planning-type documents
ci.yml Quick Checks; make pre-commit
check_no_tokio_io_uring.sh
ci-gate
Keeps tokio's io-uring backend disabled
ci.yml Quick Checks
check_s3s_footprint.sh
ci-gate
Lower-only ratchet freezing the direct s3s surface ahead of the s3gate migration
ci.yml Quick Checks; make pre-commit
check_unsafe_code_allowances.sh
ci-gate
Unsafe-code allowance ledger guard
ci.yml Quick Checks
layer-dependency-baseline.txt
ci-gate (data)
Committed baseline consumed by check_layer_dependencies.sh
arch-checks skill
static.sh
ci-gate
Static-build helper executed inside image builds
Dockerfile.source, Dockerfile.decommission-local
helm_chart_version.sh
ci-gate
Keeps the Helm chart version in sync with the release
helm-package.yml
test_helm_templates.sh
ci-gate
Helm template rendering test
helm-package.yml
Test & e2e runners
Entry
Status
Purpose
Wiring / docs
e2e-run.sh
ci-gate
Boots a rustfs server and runs the s3s-e2e black-box conformance tool against it
ci.yml e2e-tests jobs; docs/testing/README.md
run_ecstore_validation_suite.sh
dev-tool
ecstore black-box validation suite (quick/full/destructive/fuzz profiles)
Moved 2026-07 (backlog#1153 infra-13) after a whole-tree reference census:
each entry had zero references from CI, Makefiles, docs, or code — or was
referenced only by other scripts in this same archived set. Reasons:
Entry
Was
validate_issue_785_list_objects.sh
One-shot issue validation (list-objects series)
validate_issue_786_list_objects.sh
One-shot issue validation (list-objects series)
validate_issue_787_list_quorum.sh
One-shot issue validation (list-quorum)
validate_issue_841_list_objects_observability.sh
One-shot issue validation (list observability)
validate_issue_1365_docker.sh
One-shot issue validation (docker repro)
validate_issue_2723_site_replication.sh
One-shot issue validation (site replication)
validate_issue_3031_docker.sh
One-shot issue validation (docker repro)
run_issue712_deeper_zero_copy_put_with_capture.sh
One-shot perf capture for backlog#712
run_issue797_local_4node_16disk_ab.sh
One-shot 4-node/16-disk A/B for backlog#797
run_issue_2573_acceptance.sh
One-shot acceptance run for issue #2573
run_issue_2941_perf_capture.sh
One-shot perf capture for issue #2941
run_put_large_stage_breakdown.sh
backlog#706 large-PUT stage breakdown (family)
run_put_large_stage_breakdown_with_capture.sh
backlog#706 one-shot wrapper (family)
run_put_large_tuning_matrix.sh
backlog#706 tuning matrix (family)
collect_put_large_stage_breakdown_artifacts.sh
backlog#706 artifact collector (family)
analyze_put_service_metrics_deltas.py
backlog#706 metrics-delta analyzer (family)
README-stress-test.md
GET-optimization one-shot suite doc
stress-test-get-optimization.sh
GET-optimization one-shot stress test
quick-validate-get-optimization.sh
GET-optimization one-shot validation
benchmark-sf-optimization.sh
GET-optimization one-shot benchmark
prepare_gt1g_get_test_objects.sh
>1 GiB GET investigation one-shot fixture prep
run_gt1g_multipart_put_server_path_focus.sh
>1 GiB PUT investigation one-shot focus run
run_get_metrics_gate_smoke.sh
One-shot GET metrics-gate smoke
run_listobjects_verified_bench.sh
One-shot verified list-objects bench
run_object_batch_bench_abc.sh
One-shot capacity/object profile A/B/C controller
run_object_data_cache_bench.sh
One-shot GET bench for the object-data-cache rollout gate
setup-test-binaries.sh
One-shot Docker-build test binary fixture
test.sh
Ancient manual mc bucket smoke scratchpad
test_policy.json
Orphaned IAM policy fixture (hardcoded test bucket)
