Calculate target ratio for Ceph pools¶
Ceph pool target ratio defines for the Placement Group (PG) autoscaler the amount of data the pools are expected to acquire over time in relation to each other. You can set initial PG values for each Ceph pool. Otherwise, the autoscaler starts with the minimum value and scales up, causing a lot of data to move in the background.
You can allocate several pools to use the same device class, which is a solid
block of available capacity in Ceph. For example, if three pools
(kubernetes-hdd
, images-hdd
, and volumes-hdd
) are set to use the
same device class hdd
, you can set the target ratio for Ceph pools to
provide 80% of capacity to the volumes-hdd
pool and distribute the
remaining capacity evenly between the two other pools. This way, Ceph pool
target ratio instructs Ceph on when to warn that a pool is running out of free
space and, at the same time, instructs Ceph on how many placement groups Ceph
should allocate/autoscale for a pool for better data distribution.
Ceph pool target ratio is not a constant value and you can change it according to new capacity plans. Once you specify target ratio, if the PG number of a pool scales, other pools with specified target ratio will automatically scale accordingly.
For details, see Ceph Documentation: Autoscaling Placement Groups.
To calculate target ratio for each Ceph pool:
Define raw capacity of the entire storage by device class:
kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o name) -- ceph df
For illustration purposes, the procedure below uses raw capacity of 185 TB or 189440 GB.
Design Ceph pools with the considered device class upper bounds of the possible capacity. For example, consider the
hdd
device class that contains the following pools:The
kubernetes-hdd
pool will contain not more than 2048 GB.The
images-hdd
pool will contain not more than 2048 GB.The
volumes-hdd
pool will contain 50 GB per VM. The upper bound of used VMs on the cloud is 204, the pool replicated size is3
. Therefore, calculate the upper bounds forvolumes-hdd
:50 GB per VM * 204 VMs * 3 replicas = 30600 GB
The
backup-hdd
pool can be calculated as a relative ofvolumes-hdd
. For example, 1volumes-hdd
storage unit per 5backup-hdd
units.The
vms-hdd
is a pool for ephemeral storage Copy on Write (CoW). We recommend designing the amount of ephemeral data it should store. For example purposes, we use 500 GB. However, in reality, despite the CoW data reduction, this value is very optimistic.
Note
If
dataPool
is replicated and Ceph Object Store is planned for intensive use, also calculate upper bounds fordataPool
.Calculate target ratio for each considered pool. For example:
¶ Pools upper bounds
Pools capacity
kubernetes-hdd
= 2048 GBimages-hdd
= 2048 GBvolumes-hdd
= 30600 GBbackup-hdd
= 30600 GB * 5 = 153000 GBvms-hdd
= 500 GB
Summary capacity = 188196 GB
Total raw capacity = 189440 GB
Calculate pools fit factor using the (total raw capacity) / (pools summary capacity) formula. For example:
pools fit factor = 189440 / 188196 = 1.0066
Calculate pools upper bounds size using the (pool upper bounds) * (pools fit factor) formula. For example:
kubernetes-hdd = 2048 GB * 1.0066 = 2061.5168 GB images-hdd = 2048 GB * 1.0066 = 2061.5168 GB volumes-hdd = 30600 GB * 1.0066 = 30801.96 GB backup-hdd = 153000 GB * 1.0066 = 154009.8 GB vms-hdd = 500 GB * 1.0066 = 503.3 GB
Calculate pool target ratio using the (pool upper bounds) * 100 / (total raw capacity) formula. For example:
kubernetes-hdd = 2061.5168 GB * 100 / 189440 GB = 1.088 images-hdd = 2061.5168 GB * 100 / 189440 GB = 1.088 volumes-hdd = 30801.96 GB * 100 / 189440 GB = 16.259 backup-hdd = 154009.8 GB * 100 / 189440 GB = 81.297 vms-hdd = 503.3 GB * 100 / 189440 GB = 0.266
If required, calculate the target ratio for erasure-coded pools.
Due to erasure-coded pools splitting each object into
K
data parts andM
coding parts, the total used storage for each object is less than that in replicated pools. Indeed,M
is equal to the number of OSDs that can be missing from the cluster without the cluster experiencing data loss. This means that planned data is stored with an efficiency of(K+M)/2
on the Ceph cluster. For example, if an erasure-coded data pool withK=2, M=2
planned capacity is 200 GB, then the total used capacity is200*(2+2)/2
, which is 400 GB.Open the
KaasCephCluster
CR of a managed cluster for editing:kubectl edit kaascephcluster -n <managedClusterProjectName>
Substitute
<managedClusterProjectName>
with the corresponding value.In the
spec.cephClusterSpec.pools
section, specify the calculated relatives astargetSizeRatio
for each considered replicated pool. For example:spec: cephClusterSpec: pools: - name: kubernetes deviceClass: hdd ... replicated: size: 3 targetSizeRatio: 1.088 - name: images deviceClass: hdd ... replicated: size: 3 targetSizeRatio: 1.088 - name: volumes deviceClass: hdd ... replicated: size: 3 targetSizeRatio: 16.259 - name: backup deviceClass: hdd ... replicated: size: 3 targetSizeRatio: 81.297 - name: vms deviceClass: hdd ... replicated: size: 3 targetSizeRatio: 0.266
If Ceph Object Store
dataPool
isreplicated
and a proper value is calculated, also specify it:spec: cephClusterSpec: objectStorage: rgw: name: rgw-store ... dataPool: deviceClass: hdd ... replicated: size: 3 targetSizeRatio: <relative>
In the
spec.cephClusterSpec.pools
section, specify the calculated relatives asparameters.target_size_ratio
for each considered erasure-coded pool. For example:Note
The
parameters
section is a key-value mapping where the value is of the string type and should be quoted.spec: cephClusterSpec: pools: - name: ec-pool deviceClass: hdd ... parameters: target_size_ratio: "<relative>"
If Ceph Object Store
dataPool
iserasure-coded
and a proper value is calculated, also specify it:spec: cephClusterSpec: objectStorage: rgw: name: rgw-store ... dataPool: deviceClass: hdd ... parameters: target_size_ratio: "<relative>"
Verify that all target ratio has been successfully applied to the Ceph cluster:
kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o name) -- ceph osd pool autoscale-status
Example of system response:
POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE device_health_metrics 0 2.0 149.9G 0.0000 1.0 1 on kubernetes-hdd 2068 2.0 149.9G 0.0000 1.088 1.0885 1.0 32 on volumes-hdd 19 2.0 149.9G 0.0000 16.259 16.2591 1.0 256 on vms-hdd 19 2.0 149.9G 0.0000 0.266 0.2661 1.0 128 on backup-hdd 19 2.0 149.9G 0.0000 81.297 81.2972 1.0 256 on images-hdd 888.8M 2.0 149.9G 0.0116 1.088 1.0881 1.0 32 on
Optional. Repeat the steps above for other device classes.