fran/rustfs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5fbd49a8008fc97b645ca045f7944d31eded3af4
rustfs/helm/rustfs/values.yaml
T
RohmilchkaeseGitHubcxymdshousemecxymds
 8bcffd8a04 feat(helm): support multiple server pools for capacity expansion (#3325) 
* feat(helm): support multiple server pools for capacity expansion

The server already understands multiple pools (RUSTFS_VOLUMES is split on
spaces, one pool expression each; rc admin pool/expand/rebalance/
decommission exist), but the chart could only render a single StatefulSet.

Add an opt-in pools mode:

- pools.enabled=false (default) renders byte-identical output to the
  current chart (verified against five values variants)
- pools.list renders one StatefulSet per entry; entries may set
  replicaCount (4 or 16) and a storageclass block, everything else is
  inherited from the top-level values
- pool 0 keeps the legacy StatefulSet/pod/PVC names and its (immutable)
  selector, so existing single-pool deployments can be expanded in place
  without renaming or data loss; additional pools render as
  <fullname>-poolN with a rustfs.com/pool label in their selector
- RUSTFS_VOLUMES and RUSTFS_SERVER_DOMAINS enumerate every pool
- pod anti-affinity is scoped per pool so pools may share nodes while
  each pool still spreads its own pods across distinct nodes
- template validation fails fast on unsupported replicaCount, an empty
  pools.list, or pools in standalone mode
- pools are append-only by design (list index = identity), documented in
  values.yaml and the README

* fix(helm): truncate pool names DNS-safely, document PDB pool scope

Truncate <fullname>-poolN to 63 chars by shortening the base name rather
than the suffix, so the pool index always survives and two pools of a
max-length release cannot collide. Document that the single
PodDisruptionBudget deliberately spans all pools.

* fix(helm): pool-mode anti-affinity must be preferred, not required

Found by live-testing in-place expansion on a 5-node cluster (4-replica
pool 0 + 4-replica pool 1): with requiredDuringScheduling anti-affinity
the expansion deadlocks in a cycle that cannot self-heal -

  1. the not-yet-rolled pool-0 pods still carry the unscoped required
     rule and repel every rustfs pod from their nodes, so part of pool 1
     stays Pending (no IP, no headless-DNS record);
  2. every pod that already has the new RUSTFS_VOLUMES exits fatally on
     'failed to lookup address information' for those Pending peers;
  3. the StatefulSet rolling update is gated on the crashing pod going
     Ready, so the remaining pool-0 pods never roll and keep repelling.

Because StatefulSets assign revisions per ordinal, even a subsequent
template fix cannot rescue a wedged rollout (Pending ordinals are only
recreated with the new template after the higher ordinals go Ready) -
the new pool's StatefulSet has to be recreated. Shipping the soft
affinity from the start avoids the state entirely; expansion was
re-tested end-to-end with it and converges.

Pool-mode rendering only - pools.enabled=false still renders the
existing required anti-affinity byte-identically.

---------

Co-authored-by: cxymds <[email protected]>
Co-authored-by: houseme <[email protected]>
Co-authored-by: cxymds <[email protected]>
2026-07-09 07:58:06 +00:00

401 lines
15 KiB
YAML
Raw Blame History

# Default values for rustfs.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
# This will set the replicaset count more information can be found here: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
replicaCount: 4
# Kubernetes cluster DNS domain used to build in-cluster FQDNs for
# RUSTFS_VOLUMES (distributed mode) and mTLS server certificate SANs.
# Override this only if your cluster does not use the default `cluster.local`
# (for example, on-premise clusters configured with a custom domain).
# Provide the DNS root only, without a `svc.` prefix or leading/trailing dots.
clusterDomain: cluster.local
# This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/
image:
rustfs: # This sets the rustfs image repository and tag.
repository: rustfs/rustfs
# This sets the pull policy for images.
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: ""
initImage: # This sets the init container image repository and tag.
repository: busybox
pullPolicy: IfNotPresent
tag: "stable"
# This is for the secrets for pulling an image from a private repository more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
imagePullSecrets: []
imageRegistryCredentials:
enabled: false
registry: ""
username: ""
password: ""
email: ""
# This is to override the chart name.
nameOverride: ""
fullnameOverride: ""
mode:
standalone:
enabled: false
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
existingClaim:
dataClaim: ""
logsClaim: ""
distributed:
enabled: true
# Server pools for horizontal capacity expansion (distributed mode only).
# Disabled by default: the chart then behaves exactly like the classic
# single-pool layout driven by the top-level replicaCount/storageclass.
#
# With pools.enabled=true, one StatefulSet is rendered per entry of
# pools.list and RUSTFS_VOLUMES contains every pool's volume expression
# (space separated), which is how the server discovers multiple pools.
# Each entry may set replicaCount (4 or 16) and/or a storageclass block;
# omitted fields inherit the top-level values.
#
# IMPORTANT:
# - Pools are append-only. The list index determines the StatefulSet name
# (pool 0 keeps the legacy name so existing single-pool deployments can be
# expanded in place); never remove or reorder entries. To retire a pool,
# use `rc admin decommission` first.
# - After adding a pool, run `rc admin rebalance start <alias>` to spread
# existing data across the new capacity.
pools:
enabled: false
list: []
# Example: expand an existing 4-node deployment with a second, larger pool:
# list:
# - {} # pool 0: inherits top-level values, keeps existing PVCs
# - replicaCount: 4 # pool 1: new capacity
# storageclass:
# dataStorageSize: 10Gi
secret:
existingSecret: ""
# SECURITY: rendering fails by default unless one of the following is true:
# 1. `secret.existingSecret` names a Kubernetes Secret you control, or
# 2. `secret.rustfs.access_key` and `secret.rustfs.secret_key` are both
# set to non-empty, non-default values, or
# 3. `secret.allowInsecureDefaults: true` is set (only for local dev).
# This prevents accidental deployment with the well-known default
# `rustfsadmin/rustfsadmin` credentials.
allowInsecureDefaults: false
rustfs:
access_key: ""
secret_key: ""
config:
rustfs:
# Examples
# volumes: "/data/rustfs0,/data/rustfs1,/data/rustfs2,/data/rustfs3"
# volumes: "http://rustfs-{0...3}.rustfs-headless:9000/data/rustfs{0...3}"
volumes: ""
address: ":9000"
console_enable: "true"
console_address: ":9001"
log_level: "info"
region: "us-east-1"
obs_log_directory: "/logs" # Set to "" to disable log PVCs and mounts.
obs_environment: "development"
# Optionally enable support for virtual-hosted-style requests.
# See more information: https://docs.rustfs.com/integration/virtual.html
domains: "" # e.g. "example.com"
ec:
storage_class_standard: "EC:4" # Storage class for standard storage class in erasure coding mode, default is "STANDARD"
log_rotation: # Specify log rotation settings
# size: 100 # Default value: 100 MB
# time: hour # Default value: hour, eg: day,hour,minute,second
# keep_files: 30 # number of rotated log files to keep
scanner:
# Scanner speed preset: fastest|fast|default|slow|slowest
speed: ""
# Override scanner sleep multiplier (0-10000)
delay: ""
# Override maximum scanner sleep in seconds
max_wait_secs: ""
# Override scanner cycle interval in seconds
cycle_secs: ""
# Override start delay in seconds (optional)
start_delay_secs: ""
# Cap one scanner cycle's runtime in seconds (0 disables)
cycle_max_duration_secs: ""
# Cap objects processed by one scanner cycle (0 disables)
cycle_max_objects: ""
# Cap directories entered by one scanner cycle (0 disables)
cycle_max_directories: ""
# Periodic deep bitrot cycle in seconds; false/off/no/disabled disables
bitrot_cycle_secs: ""
# Enable/disable scanner sleeps for throttling
idle_mode: ""
# Timeout for scanner cache saves in seconds (minimum 1 second)
cache_save_timeout_secs: ""
# Cap concurrent scanner set tasks (0 keeps topology-derived concurrency)
max_concurrent_set_scans: ""
# Cap concurrent scanner disk bucket walks per set (0 keeps disk-count-derived concurrency)
max_concurrent_disk_scans: ""
# Yield to the async runtime every N scanned objects (0 disables extra yield)
yield_every_n_objects: ""
# Version count threshold for scanner alerts
alert_excess_versions: ""
# Retained version byte threshold for scanner alerts
alert_excess_version_size: ""
# Direct subfolder threshold for scanner alerts
alert_excess_folders: ""
# Drive timeout profile preset: default|high_latency
# - default: keep current timeout defaults.
# - high_latency: use higher timeout defaults for scanner-sensitive operations.
timeout_profile: ""
obs_endpoint:
enabled: false # If true, rustfs will export metrics, traces, logs and profiling data to the specified OTLP endpoints. If false, the individual settings for metrics, traces, logs and profiling endpoints will be ignored and all data will not be exported.
base_endpoint: "" #Root OTLP/HTTP endpoint, e.g. http://otel-collector:4318
use_stdout: false # If true, rustfs will also export logs to stdout in addition to the OTLP logs endpoint.
metrics:
enabled: false
endpoint: "" # If specified, rustfs will export metrics to this OTLP endpoint. e.g. "http://localhost:4318/v1/metrics"
trace:
enabled: false
endpoint: "" # If specified, rustfs will export traces to this OTLP endpoint. e.g. "http://localhost:4318/v1/traces"
logs:
enabled: false
endpoint: "" # If specified, rustfs will export logs to this OTLP endpoint. e.g. "http://localhost:4318/v1/logs"
profiling:
enabled: false
endpoint: "" # If specified, rustfs will export profiling data to this endpoint. e.g. "http://localhost:6060/debug/pprof/profile"
kms:
enabled: false
type: "vault" # Only Support vault currently.
vault:
vault_backend: "" # Only support vault kv2 and vault transit.
vault_address: ""
vault_token: ""
vault_mount_path: ""
default_key: ""
extraEnv: [] # This is for setting extra environment variables in the rustfs container. It should be a list of key value pairs. For example:
# extraEnv:
# - name: RUSTFS_ERASURE_SET_DRIVE_COUNT
# value: "16"
# - name: RUSTFS_STORAGE_CLASS_STANDARD
# value: "EC:4"
extraVolumes: []
extraVolumeMounts: []
# This section builds out the service account more information can be found here: https://kubernetes.io/docs/concepts/security/service-accounts/
serviceAccount:
# Specifies whether a service account should be created
create: true
# Automatically mount a ServiceAccount's API credentials?
automount: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
# This is for setting Kubernetes Annotations to a Pod.
# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
podAnnotations: {}
# This is for setting Kubernetes Labels to a Pod.
# For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
podLabels: {}
# Labels to add to all deployed objects
commonLabels: {}
podSecurityContext:
fsGroup: 10001
runAsUser: 10001
runAsGroup: 10001
containerSecurityContext:
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
runAsNonRoot: true
# Name of an existing PriorityClass to assign, defining the importance of the pods compared to other pods in the cluster.
# Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/
priorityClassName: ""
service:
annotations: {}
headlessAnnotations: {} # Applied to the headless Service when mode.distributed.enabled=true
traefikAnnotations: # Applied to the Service when mode.distributed.enabled=true and ingress.className=traefik
traefik.ingress.kubernetes.io/service.sticky.cookie: "true"
traefik.ingress.kubernetes.io/service.sticky.cookie.httponly: "true"
traefik.ingress.kubernetes.io/service.sticky.cookie.name: rustfs
traefik.ingress.kubernetes.io/service.sticky.cookie.samesite: none
traefik.ingress.kubernetes.io/service.sticky.cookie.secure: "true"
type: ClusterIP
# LoadBalancer-specific fields (only used when type=LoadBalancer)
loadBalancerIP: "" # Request a specific IP from the load balancer (e.g. for Cilium LB-IPAM or MetalLB)
loadBalancerClass: "" # Specify a load balancer implementation (e.g. "io.cilium/bgp", "metallb")
loadBalancerSourceRanges: [] # Restrict client IPs (e.g. ["10.0.0.0/8"])
endpoint:
port: 9000
nodePort: 32000
console:
port: 9001
nodePort: 32001
# This block is for setting up the ingress for more information can be found here: https://kubernetes.io/docs/concepts/services-networking/ingress/
ingress:
enabled: true
className: "nginx" # Specify the classname, traefik or nginx. Different classname has different annotations for session sticky.
traefikAnnotations: {} # Deprecated: use service.traefikAnnotations instead
nginxAnnotations:
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/session-cookie-expires: "3600"
nginx.ingress.kubernetes.io/session-cookie-hash: sha1
nginx.ingress.kubernetes.io/session-cookie-max-age: "3600"
nginx.ingress.kubernetes.io/session-cookie-name: rustfs
annotations: {}
customAnnotations: # Deprecated: use ingress.annotations instead
{}
hosts:
- host: example.rustfs.com
paths:
- path: /
pathType: Prefix
tls:
enabled: false # Enable tls and access rustfs via https.
certManager:
enabled: false # Enable certmanager to generate certificate for rustfs, default false.
existingSecret:
# To use an existing secret for tls certificates in the ingress enable this,
# set the name of the existing secret and make sure that the secret contains
# values for the keys "crt" and "key".
enabled: false
name: ""
# If the certmanager is not used and no existing secret is used to provide a tls certificate,
# then this certificate can be configured here. A secret will be created and used.
# Note: The following input will be base64 encoded during secret creation. There is no need
# to provide it base64 encoded here already.
crt: tls.crt
key: tls.key
gatewayApi:
enabled: false
gatewayClass: traefik # Only support for traefik and contour gatewayClass at the moment.
listeners: # Specify which listeners to create on the Gateway.
http:
name: web
port: 8000
https:
name: websecure
port: 8443
hostname: example.rustfs.com
existingGateway:
name: ""
namespace: ""
mtls:
enabled: false
clientCertPath: "/opt/tls/client_cert.pem"
clientKeyPath: "/opt/tls/client_key.pem"
existingIssuerRef:
enabled: false
name: ""
kind: ""
group: ""
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 200m
# memory: 512Mi
# requests:
# cpu: 100m
# memory: 128Mi
# This is to setup the liveness and readiness probes more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
livenessProbe:
enabled: true # omitted
httpGet:
path: /health
port: 9000
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 5
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 3
readinessProbe:
enabled: true # omitted
httpGet:
path: /health/ready
port: endpoint
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 3
nodeSelector: {}
tolerations: []
affinity:
podAntiAffinity:
enabled: true
topologyKey: kubernetes.io/hostname
nodeAffinity: {}
topologySpreadConstraints:
enabled: false
constraints: []
# - maxSkew: 1
# topologyKey: topology.kubernetes.io/zone
# whenUnsatisfiable: ScheduleAnyway
# labelSelector:
# matchLabels:
# app.kubernetes.io/name: rustfs
storageclass:
name: local-path
dataStorageSize: 256Mi
logStorageSize: 256Mi
pvcAnnotations: {}
#pvcAnnotations:
# data:
# key: value
# logs:
# key: value
pdb:
create: false
# Minimum number/percentage of pods that should remain scheduled
minAvailable: ""
# Maximum number/percentage of pods that may be made unavailable
maxUnavailable: 1
# Whether information about services should be injected into pod's environment variable
enableServiceLinks: false
extraManifests: []
Reference in New Issue View Git Blame Copy Permalink