Creating a Multi-Zone Application
Creating an application and deploying it across several zones to increase the service's availability and stability is possible in 3 Arvancloud data centers: Bamdad, Forough, and Simin. To do this, you can use both the panel and the CLI with manifests.
When creating Multi-Zone applications, note the following:
- In this deployment method, disks (Persistent Volumes) in different zones aren't synced with each other, and you need to use mechanisms at the application level for this, such as Redis and Kafka.
- In the panel, only applications without disks (stateless) can be made Multi Zone. To make stateful applications Multi Zone, you need to use kubectl or Helm.
Creating a Multi-Zone Application via the CLI
Kubernetes can automatically distribute Pods among the different nodes of a cluster. But to reduce the impact of disruptions, you may want to specify how Pods are spread across each zone or node. For example, you can specify that when you only have two Pods, they're not both placed in the same data center, or that, besides distributing Pods equally among data centers, no data center has more than 2 Pods.
The Affinity and Topology Spread Constraints concepts in Kubernetes are designed for this.
-
Anti-Affinity
In Kubernetes, Affinity / Anti-Affinity controls how Pods are placed relative to each other. To make sure similar Pods aren't placed in the same zone, you need to use Affinity. For example, consider the following Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/cloud-container-g2
operator: In
values:
- "true"
- key: topology.kubernetes.io/zone
operator: In
values:
- bamdad
- simin
- forough
containers:
- name: my-app
image: my-app-image
In the affinity section, which defines how Pods are distributed relative to each other, podAntiAffinity guarantees that Pods aren't placed together in a topology such as a zone.
-
requiredDuringSchedulingIgnoredDuringExecutionSpecifies that Anti-Affinity must be followed during scheduling, but can be ignored at runtime. For example, if a node becomes unavailable and the Pod is rescheduled, the rule doesn't need to be evaluated again. This is called Hard Anti-Affinity, and if these conditions aren't met, the Pod isn't scheduled.
On the other hand,
preferredDuringSchedulingIgnoredDuringExecutionis known as Soft Anti-Affinity. In this mode, the scheduler tries to follow the rules, but if the conditions aren't met, the Pod is still scheduled.Note that Soft Anti-Affinity provides more flexibility. With Hard Anti-Affinity, if the rule can't be followed, the application may become unavailable.
-
labelSelectorSpecifies the Pods the Anti-Affinity rule applies to.
-
topologyKey: topology.kubernetes.io/zoneSpecifies the scope of the Anti-Affinity rule. The key
"topology.kubernetes.io/zone"means Pods must be distributed at the zone level. -
Pod Topology Spread Constraints
When a node starts, a label is added to it that can include data center or zone information. If your cluster spans several zones, you can use these labels together with a concept called Pod Topology Spread Constraints to specify how Pods are distributed and placed across zones.
For example, consider the following Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: ng
name: ng
namespace: myproject
spec:
replicas: 3
selector:
matchLabels:
app: ng
template:
metadata:
creationTimestamp: null
labels:
app: ng
name: ng
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/cloud-container-g2
operator: In
values:
- "true"
- key: topology.kubernetes.io/zone
operator: In
values:
- bamdad
- simin
- forough
containers:
- image: nginx:latest
imagePullPolicy: IfNotPresent
name: ng
ports:
- containerPort: 80
name: http
protocol: TCP
resources:
limits:
cpu: "1"
ephemeral-storage: 1G
memory: 2G
requests:
cpu: "1"
ephemeral-storage: 1G
memory: 2G
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
key: role
operator: Equal
value: cloud-container-g2
topologySpreadConstraints:
- labelSelector:
matchLabels:
app: ng
name: ng
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
-
topologyKeySpecifies the level at which Pods are distributed. For example, in this example, the settings apply at the
Zonelevel. -
whenUnsatisfiableThis variable specifies what to do when the conditions aren't met. The value
DoNotSchedule, used in this example, means that if the conditions aren't met, the Pod isn't scheduled.This parameter can also accept
ScheduleAnyway. In this mode, the Pod starts even if the conditions aren't met. -
maxSkewSpecifies the maximum allowed difference between the number of Pods in the domain defined in
topologyKey, which here is the zone. That is, in this example, the number of Pods in each zone can be at most one more than in another zone. -
labelSelectorUsed to select Pods based on their labels. In effect, it specifies which Pods these rules apply to. For example, in this example, Pods with the labels
app: ngandname: ngare considered.
Example: Setting Up a Multi-Zone Redis Cluster with an Operator
Save the following file with a name of your choice, such as redis-cluster.yaml:
apiVersion: redis.redis.opstreelabs.in/v1beta2
kind: RedisCluster
metadata:
name: redis-cluster
labels:
app.kubernetes.io/name: redis-cluster
spec:
clusterSize: 3
persistenceEnabled: true
clusterVersion: v7
redisLeader:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "topology.kubernetes.io/zone"
operator: In
values:
- "simin"
- "forough"
- "bamdad"
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: redis-cluster
topologyKey: "topology.kubernetes.io/zone"
replicas: 3
redisFollower:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "topology.kubernetes.io/zone"
operator: In
values:
- "simin"
- "forough"
- "bamdad"
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: redis-cluster
topologyKey: "topology.kubernetes.io/zone"
replicas: 3
redisExporter:
enabled: false
image: "quay.io/opstree/redis-exporter:v1.44.0"
imagePullPolicy: "IfNotPresent"
kubernetesConfig:
image: "quay.io/opstree/redis:v7.0.12"
imagePullPolicy: "IfNotPresent"
storage:
nodeConfVolume: true
nodeConfVolumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
podSecurityContext:
fsGroup: 1000
runAsUser: 1000
Then run the following command to deploy it:
kubectl apply -f redis-cluster.yaml -n [namespace]
Replace [namespace] with your project's name in the Arvancloud Cloud Container.
This part of the manifest is written to distribute the redisLeader nodes among 3 zones: Simin, Forough, and Bamdad:
redisLeader:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "topology.kubernetes.io/zone"
operator: In
values:
- "simin"
- "forough"
- "bamdad"
Also, in this part, podAntiAffinity keeps Pods from being placed in the same zone, so each node sits in one zone:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: redis-cluster
topologyKey: "topology.kubernetes.io/zone"
This part of the manifest is written to distribute the redisFollower nodes among 3 zones: Simin, Forough, and Bamdad:
redisFollower:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "topology.kubernetes.io/zone"
operator: In
values:
- "simin"
- "forough"
- "bamdad"
Also, in this part, podAntiAffinity keeps follower Pods from being placed in the same zone, so each node sits in one zone:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: redis-cluster
topologyKey: "topology.kubernetes.io/zone"
For more details and examples, see the Redis Operator documentation.
Creating a Multi-Zone Application via the Panel
When creating an application with a Docker image in the panel, you can set the details of the application's deployment zone in the Settings tab.
With the Docker image method, only applications without disks (stateless) can use Multi Zone. To make stateful applications Multi Zone, you can use kubectl or Helm.
-
Automatic zone selection:
When you enable this feature, the zone the application is deployed in is selected automatically. By disabling it, you can choose the zone(s) you want yourself.
-
Number of replicas:
If you've disabled automatic zone selection, you can set how many instances of your Pod should be created in this section.
-
MaxSkew:
This value specifies the maximum difference in replicas between zones.
-
-
Fallback behavior
-
DoNotSchedule:
This option tells the scheduler not to schedule the Pod on a new zone if one of the selected zones becomes unavailable for any reason.
This mode is used when strictly following the specified rules is more important, and the Pod shouldn't be scheduled until the conditions are met.
-
ScheduleAnyway:
This option tells the scheduler to schedule the Pod in any case, even if the specified rules aren't followed. Also, if a zone becomes unavailable, the Pod is scheduled on a zone that already has replicas. In other words, this method may cause an unbalanced distribution of Pods.
This mode is used when scheduling flexibility is more important than a balanced distribution of Pods.
-