9ddb30139d052d2fa79983edddd979aac7fe98ec
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Zhengchao AnandGitHub
|
2c113542f8 |
docs(architecture): normative erasure-coding algorithm and on-disk compatibility contract (#4999)
* docs(architecture): add normative erasure-coding algorithm and on-disk compatibility contract Adds docs/architecture/erasure-coding.md as the source of truth for how RustFS erasure-codes, stores, reads, reconstructs, and heals user data, and the on-disk (xl.meta) / decode compatibility contract every future change must preserve. Grounded in the baseline (main) implementation with file:line anchors; cross-links (does not duplicate) the existing placement, MinIO-format-compat, layout-boundary, decommission, and tier-ILM docs, and the AGENTS.md cross-cutting invariants. Covers: Reed-Solomon over GF(2^8) modern vs GF(2^16) legacy backends and how each is selected; pool/set/drive geometry with the 2..=16 set-size and per-pool parity invariants; the key-derived distribution permutation (1..=N); 1 MiB block size and the modern/legacy shard-size formulas; HighwayHash256S interleaved bitrot layout; the full xl.meta container/header/version-body schema and internal dual-key convention; write/read/heal quorum rules; and, newly codified as a first-class contract, the decode-tolerance invariants (nil-UUID/epoch-mod_time to None, skip unknown fields, hard-guard only length-critical arrays, tolerate the negative actual_size compressed sentinel, tolerate a malformed transitioned-versionID). Linked from the architecture README under "Contracts & invariants". Docs-only; check_doc_paths.sh passes. * docs(architecture): anchor erasure spec by symbol name, not line numbers Line numbers rot as code changes and are not validated by check_doc_paths.sh, so they would silently mislead the very code changes this normative spec is meant to guide. Replace all file:line citations with file-path + symbol-name references (functions/consts/types are greppable and rename only on deliberate changes; format byte offsets are kept). Add an explicit "references work here" note and a §13 rule that governed changes must update this spec in the same PR. * docs(architecture): correct erasure spec after multi-expert adversarial review Four independent adversarial reviewers fact-checked every claim against the code. Fixes: - CRITICAL (found independently by two reviewers): §1 mislabeled the GF(2^16) reed-solomon-simd "legacy" backend as the reader for "older MinIO-lineage format". It is the opposite — that backend serves RustFS's own older main-branch (rmp_serde, uses_legacy_checksum) objects; MinIO-migrated data uses the same rs-vandermonde GF(2^8) scheme and is decoded by the modern backend. The old wording contradicted §1's own MinIO-interop line, §12, minio-file-format-compat.md, and the source comments, and would have misled the highest-stakes decode-routing decision. - §2.1/§12: set size 2..=16 holds for multi-drive layouts; single-drive deployments run at N=1 (parity 0) outside is_valid_set_size. - §2.2: validate_parity_inner enforces parity <= N/2 only for N > 2 (user storage-class parity flows through it); the standalone validate_parity is unconditional but only applied to the resolved default. - §6.2/§12: header format is dispatched by header_ver; array length (4/5/7) is a per-version validation, not the discriminator. - §6.3: the part-array length guard applies on the all_parts decode path. - §6.5: get_bytes matches only the two canonical lowercase keys; only is_internal_key/get_str are case-insensitive. - §6.5: transitioned-versionID — state the load-bearing invariant (non-16-byte decodes to None, never fatal); string-form recovery is optional, and transitioned-xl.meta interop is out of scope. - §11: typo RustSF -> RustFS. All other claims across §1-§14 were verified accurate against code (distribution formula, quorum formulas, on-disk key set/endianness/markers, decode-tolerance invariants, version anchors, standard references). |
||
|
Zhengchao AnandGitHub
|
f40abbb9f2 |
docs(architecture): unify per-object generation authority (#4912)
Add the shared design/contract document for backlog #1326: a single per-object generation authority (the #1312 fencing epoch) that spans commit fencing, read lease, prepared pool read, quota reservation, and old-dir GC. Pins the five cross-cutting constraints once - RPC signature binding with server-side nonce enforcement, xl.meta encoding contract (no meta_ver bump, no positional FileInfo field, internal metadata map under the dual-key contract), proto3 optional presence, mixed-version fallback direction, and cluster-level capability negotiation - so the implementation sub-issues follow them rather than each re-deciding. No product code changes. |
||
|
Zhengchao AnandGitHub
|
c4c198670d |
docs: remove agent-generated planning docs and forbid committing them (#4771)
docs: remove agent-generated planning docs, forbid committing them Delete one-shot planning/progress artifacts that were checked into the tree: the 14 superpowers plan/tracker docs under docs/superpowers/plans/, plus issue-scoped implementation plans, optimization conclusions, and dated benchmark-result snapshots under docs/ (issue-4003 ListObjectsV2 plans, get-small-file conclusion, issue824/issue829 benchmark results, issue-713 >1GiB GET baseline summary and ops guide). Codify the rule so they do not come back: - .gitignore drops the docs/superpowers whitelist, so anything new under docs/ stays ignored unless force-added. - AGENTS.md gains an explicit 'do not commit planning-type documents' rule scoping version control to the durable architecture/operations/testing sets. - docs/architecture/README.md, overview.md, arch-checks SKILL.md, and check_doc_paths.sh drop their references to the removed archive. |
||
|
Zhengchao AnandGitHub
|
f63af3df63 |
chore: retire completed-migration scaffolding, wire orphaned boundary check (#4719)
The ecstore/global-state migrations are done (backlog#815, #939, #1052 all closed). Review of every migration-era test/gate measure found three things actually retirable or broken — everything else is a live anti-regression guard and stays. Remove: - scripts/check_metrics_migration_refs.sh — guards a migration that finished: rustfs_metrics:: has zero hits, the metrics crate no longer exists, and the script was never wired into CI or make (only reference was one line in config-model-boundary-adr.md, also removed). - crates/obs init_metrics_collectors — the "backward-compatible alias kept during migration" the removed script was guarding. Zero callers; pure delegate to init_metrics_runtime. Archive (docs/superpowers/plans/, continuing the 2026-07 convention, with the standard archived banner): - startup-timeline.md, scheduler-baseline.md, profiling-numa-capability-inventory.md, kms-development-defaults-inventory.md — one-shot snapshots whose only consumer is the already-archived migration-progress ledger (their same-dir links there start resolving again after the move); zero script pins; fed the closed backlog#660/#665 architecture-review ledger. Fixed the one outbound link (startup-timeline -> readiness-matrix) that the move would have broken — check_doc_paths.sh deliberately does not scan plans/, so nothing else would have caught it. Wire (found orphaned by the same review): - scripts/check_extension_schema_boundaries.sh guards a live contract crate but was never invoked anywhere. Add lint-fmt.mak target, include in pre-commit/pre-pr/dev-check, add ci.yml Quick Checks step (job already installs ripgrep), sync the CONTRIBUTING.md enumerated list, and harden the script against a silently-passing rg probe when src/ is missing. Keep (verified live, documented so the next cleanup pass does not repeat this analysis): - scripts/check_architecture_migration_rules.sh — added a header stating it is a permanent boundary guard, not retirable migration scaffolding; 'migration' in the name is historical. - check_migration_gate_count.sh + floor, delete-marker e2e proof, all pinned docs, compat-cleanup-register sync, remaining inventories (referenced by live docs). Verification: all 7 guard scripts pass, actionlint clean, cargo check --workspace (excl e2e) clean, cargo fmt --check clean. Adversarially reviewed by two independent skeptic passes; their 7 findings (alias left behind, broken outbound link, missing banners, wrong backlog attribution, CONTRIBUTING drift, rg exit-2 hole, missing header rationale) are all folded in. |
||
|
Zhengchao AnandGitHub
|
0271abc14d |
docs: add MinIO compatibility router + file-format docs (#4328)
Add two durable architecture references grounded in current code: - minio-rustfs-router-compatibility.md: MinIO cmd/api-router.go (S3 object/bucket) and cmd/admin-router.go (admin /v3, /v4) vs RustFS implementation status, with per-row landing points and a gaps-only checklist of still-missing admin endpoints (profiling, healthinfo, LDAP IDP config, replication diff, MRF, batch, locks, speedtest, log stream, top, trace). - minio-file-format-compat.md: xl.meta (meta_ver 3 write, <=3 read, XL2 magic, rs-vandermonde, HighwayHash256 bitrot, inline data) and .metadata.bin bucket-metadata interop matrix for the backlog#580 items, plus old-RustFS -> new migration and a phased plan. Cross-links s3-compatibility-matrix.md and admin-route-action-snapshot.md and indexes both docs in docs/architecture/README.md. Refs rustfs/backlog#596 rustfs/backlog#603 rustfs/backlog#580 |
||
|
|
d1db9a10cd |
chore(docs): refresh agent docs, guard doc paths, archive plans (#4203)
Agent-instruction and architecture docs had drifted from the code: - CLAUDE.md: slim to commands + pointers; fix wrong claim that `make pre-commit` is the full pre-PR gate (that is `make pre-pr`); drop stale pre-#3929 file paths and merged bug narratives - AGENTS.md: drop dead `rust-refactor-helper` skill rule and the hand-maintained (already stale) scoped-AGENTS index; link architecture docs from Sources of Truth - .github/AGENTS.md: replace the outdated copied CI command matrix with a pointer to ci.yml - crates/AGENTS.md: merge duplicated Testing sections - ARCHITECTURE.md: resolve the utils->config contradiction (edges are removed), mark volatile counts as snapshots, fix a bad path - docs/architecture: add README router; move one-shot plans/trackers (rebalance-decommission phases, migration-progress ledger, PR template) to docs/superpowers/plans with archive headers; fix stale source paths in kept inventories (core/sets.rs, core/pools.rs, store/mod.rs, startup_* split from #3671) - docs/operations/tier-ilm-debugging.md: extracted tier debugging playbook with corrected paths - scripts/check_doc_paths.sh: new guard failing pre-commit/pre-pr when instruction/architecture docs reference nonexistent file paths - .claude/skills: add tier-debug and arch-checks repo skills; .gitignore now keeps .claude/skills and docs/operations committable Verification: ./scripts/check_doc_paths.sh, ./scripts/check_architecture_migration_rules.sh (both pass) Co-authored-by: Claude Fable 5 <[email protected]> |