Large clusters

This section describes a validated MOSK cluster architecture that is capable of handling 10,000 instances under a single control plane.

Hardware characteristics

Node roles layout

Role

Nodes count

Server specification

Management cluster Kubernetes nodes

3

  • 16 vCPU 3.4 GHz

  • 32 GB RAM

  • System disk (/): 1 x NVMe or SSD 500 GB

  • Management LVP: 1 x NVMe or SSD 960 GB on a separate physical device

  • 2 x 10 Gbps NICs

MOSK cluster Kubernetes master nodes

3

  • 16 vCPU 3.4 GHz

  • 32 GB RAM

  • System disk (/): 1 x NVMe or SSD 500 GB

  • LVP is not required. OpenStack databases and StackLight run on other nodes. If local PVs are required, add a separate SSD of 480–960 GB. Do not place that LVP disk on the system disk.

  • 2 x 10 Gbps NICs

OpenStack controller nodes

5

  • 64 vCPU 2.5 GHz

  • 256 RAM

  • System disk (/): 1 x NVMe or SSD 500 GB

  • OpenStack LVP: 2 x 3.8 TB NVMe drives on separate physical devices. Do not place the host operating system on these NVMe drives. StackLight LVP is not used here because StackLight runs on dedicated nodes.

  • 2 x 25 Gbps NICs

OpenStack compute and storage nodes

Up to 500 total

  • 64 vCPU 2.5 GHz

  • 256 RAM

  • System disk (/): 1 x NVMe or SSD 500 GB

  • Ceph OSD data: 2 x 3.8 TB NVMe drives, wiped and kept separate from the system disk

  • LVP is not required. If custom workloads need LVP, place it on another physical device, not on the system disk or the OSD drives.

  • 2 x 25 Gbps NICs

StackLight nodes

3

  • 64 vCPU 2.5 GHz

  • 256 RAM

  • System disk (/): 1 x NVMe or SSD 500 GB

  • StackLight LVP: 2 x 3.8 TB NVMe drives on separate physical devices. Do not place the host operating system on these NVMe drives.

  • 2 x 25 Gbps NICs

Note

If a node role requires a Local Volume Provisioner (LVP) disk, place it on a separate physical device from the system disk that contains /. Mirantis does not recommend creating the LVP volume on the system disk. For disk roles and sizes, see Host operating system disk layout.

Cluster architecture

Cluster architecture

Configuration

Value

Dedicated StackLight nodes

Yes

Dedicated Ceph storage nodes

Yes

Dedicated control plane Kubernetes nodes

Yes

Dedicated OpenStack gateway nodes

No, collocated with OpenStack controller nodes

OpenStack networking backend

Open vSwitch, no Distributed Virtual Router

Cluster size in the OpenStackDeployment CR

medium

Cluster validation

The architecture validation is performed by means of simultaneous creation of multiple OpenStack resources of various types and execution of functional tests against each resource. The amount of resources hosted in the cluster at the moment when a certain threshold of non-operational resources starts being observed, is described below as cluster capacity limit.

Note

A successfully created resource has the Active status in the API and passes the functional tests, for example, its floating IP address is accessible. The MOSK cluster is considered to be able to handle the created resources if it successfully performs the LCM operations including the OpenStack services restart, both on the control and data plane.

Note

The key limiting factor for creating more OpenStack objects in this illustrative setup is hardware resources (vCPU and RAM) available on the compute nodes.

OpenStack resource capacity limits

OpenStack resource

Limit

Instances

11101

Network ports - instances

37337

Network ports - service (avg. per gateway node)

3517

Volumes

2784

Routers

2448

Networks

3383

Orchestration stacks

2419

Hardware resources utilization

Consumed hardware resources by a filled up cluster in the idle state

Node role

Load average

vCPU

RAM in GB

OpenStack controller + gateway

10

10

100

OpenStack compute

30

25

160

Ceph storage

2

2

15

StackLight

10

8

102

Kubernetes master

10

6

13