Tune OpenSearch performance for the bare metal provider

The following hardware recommendations and software settings apply for better OpenSearch performance in a baremetal-based Container Cloud cluster.

To tune OpenSearch performance:

  1. Depending on your cluster size, set the required disk and CPU size along with memory limit and heap size.

    Heap size is calculated in StackLight as ⅘ of the specified memory limit. If the calculated heap size exceeds 32 GB, slightly crossing this threshold causes significant waste of memory due to loss of Ordinary Object Pointers (OOPS) compression, which allows storing 64-bit pointers in 32-bits.

    Since Cluster releases 17.0.0, 16.0.0, and 14.1.0, to prevent this behavior, for the memory limit in the 31-50 GB range, the heap size is set to fixed 31 GB using the enforceOopsCompression parameter, which is enabled by default. For details, see Enforce OOPS compression. Exceeding the range causes loss of benefit of OOPS compression, so the ⅘ formula applies again.

    OpenSearch is write-heavy, so SSD is preferable as a disk type.

    Hardware recommendations for OpenSearch

    Cluster size

    Memory limit (GB)

    Heap size (GB)

    CPU (# of cores)

    Small

    16

    12.8

    2

    Medium

    32

    25.6

    4

    Large

    64

    51.2

    8

    To configure hardware settings for OpenSearch, refer to Resource limits in the Configure StackLight section.

  2. Configure the maximum count of mmap files. OpenSearch uses mmapfs to map shards stored on disk, which is set to 65530 by default.

    To verify max_map_count:

    sysctl -n vm.max_map_count

    To increase max_map_count, follow the Create a custom host profile procedure.

    Example configuration:

    kernelParameters:
  sysctl:
    vm.max_map_count: "<value>"

    Extended retention periods, which depend on open shards, require increasing this value significantly. For example, to 262144.

  3. Configure swap as it significantly degrades performance. Lower swappiness to 1 or 0 (to disable swap). For details, use the Create a custom host profile procedure.

    Example configuration:

    kernelParameters:
  sysctl:
    vm.swappiness: "<value>"

  4. Configure the kernel I/O scheduler to improve timing of disk writing operations. Change it to one of the following options:

    • none - applies the FIFO queue.

    • mq-deadline - applies three queues: FIFO read, FIFO write, and sorted.

    Changing I/O scheduling is also possible through BareMetalHostProfile. However, the specific implementation highly depends on the disk type used:

    cat /sys/block/sda/queue/scheduler

mq-deadline kyber bfq [none]