Calculate target ratio for Ceph pools

Ceph pool target ratio defines for the Placement Group (PG) autoscaler the amount of data the pools are expected to acquire over time in relation to each other. You can set initial PG values for each Ceph pool. Otherwise, the autoscaler starts with the minimum value and scales up, causing a lot of data to move in the background.

You can allocate several pools to use the same device class, which is a solid block of available capacity in Ceph. For example, if three pools (kubernetes-hdd, images-hdd, and volumes-hdd) are set to use the same device class hdd, you can set the target ratio for Ceph pools to provide 80% of capacity to the volumes-hdd pool and distribute the remaining capacity evenly between the two other pools. This way, Ceph pool target ratio instructs Ceph on when to warn that a pool is running out of free space and, at the same time, instructs Ceph on how many placement groups Ceph should allocate/autoscale for a pool for better data distribution.

Ceph pool target ratio is not a constant value and you can change it according to new capacity plans. Once you specify target ratio, if the PG number of a pool scales, other pools with specified target ratio will automatically scale accordingly.

For details, see Ceph Documentation: Autoscaling Placement Groups.

To calculate target ratio for each Ceph pool:

1. Define raw capacity of the entire storage by device class:

   kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o name) -- ceph df
   

   For illustration purposes, the procedure below uses raw capacity of 185 TB or 189440 GB.

2. Design Ceph pools with the considered device class upper bounds of the possible capacity. For example, consider the hdd device class that contains the following pools:

   • The kubernetes-hdd pool will contain not more than 2048 GB.

   • The stacklight-hdd pool will contain not more than 100000 GB.

   Note

   If dataPool is replicated and Ceph Object Store is planned for intensive use, also calculate upper bounds for dataPool.

3. Calculate target ratio for each considered pool. For example:

   Example bounds and capacity

   Pools upper bounds	Pools capacity
   kubernetes-hdd = 2048 GB stacklight-hdd = 100000 GB	Summary capacity = 102048 GB Total raw capacity = 189440 GB

   1. Calculate pools fit factor using the (total raw capacity) / (pools summary capacity) formula. For example:

      pools fit factor = 189440 / 102048 = 1.8563
      

   2. Calculate pools upper bounds size using the (pool upper bounds) * (pools fit factor) formula. For example:

      kubernetes-hdd = 2048 GB * 1.8563 = 3801.7024 GB
      stacklight-hdd = 100000 GB * 1.8563 = 185630 GB
      

   3. Calculate pool target ratio using the (pool upper bounds) * 100 / (total raw capacity) formula. For example:

      kubernetes-hdd = 3801.7024 GB * 100 / 189440 GB = 2.007
      stacklight-hdd = 185630 GB * 100 / 189440 GB = 97.989
      

4. Open the KaasCephCluster CR of a managed cluster for editing:

   kubectl edit kaascephcluster -n <managedClusterProjectName>
   

   Substitute <managedClusterProjectName> with the corresponding value.

5. In the spec.cephClusterSpec.pools section, specify the calculated relatives as targetSizeRatio for each considered pool:

   spec:
     cephClusterSpec:
       pools:
       - name: kubernetes
         deviceClass: hdd
         ...
         replicated:
           size: 3
           targetSizeRatio: 2.007
       - name: stacklight
         deviceClass: hdd
         ...
         replicated:
           size: 3
           targetSizeRatio: 97.989
   

   If Ceph Object Store dataPool is replicated and a proper value is calculated, also specify it:

   spec:
     cephClusterSpec:
       objectStorage:
         rgw:
           name: rgw-store
           ...
           dataPool:
             ...
             replicated:
               deviceClass: hdd
               size: 3
               targetSizeRatio: <relative>
   

6. Verify that all target ratio has been successfully applied to the Ceph cluster:

   kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o name) -- ceph osd pool autoscale-status
   

   Example of system response:

   POOL                   SIZE  TARGET SIZE  RATE    RAW CAPACITY  RATIO  TARGET RATIO  EFFECTIVE RATIO  BIAS  PG_NUM  NEW PG_NUM  AUTOSCALE
   device_health_metrics  0                  2.0     149.9G        0.0000                                1.0   1                   on
   kubernetes-hdd         2068               2.0     149.9G        0.0000  2.007        2.0076           1.0   32                  on
   stacklight-hdd         2068               2.0     149.9G        0.0000  97.989       97.9894          1.0   256                 on
   

7. Optional. Repeat the steps above for other device classes.