Calculate the storage retention time

Obsolete since 2.26.0 (17.1.0, 16.1.0) for OpenSearch Available since 2.22.0 and 2.23.1 (12.7.0, 11.6.0)

Caution

In Container Cloud 2.26.0 (Cluster releases 17.1.0 and 16.1.0), the storage-based log retention together with the updated proportion of available disk space replaces the estimated storage retention management in OpenSearch. For details, see Storage-based log retention strategy.

The logging.retentionTime parameter is removed from the StackLight configuration. While the Estimated Retention panel of the OpenSearch dashboard in Grafana can provide some information, it does not provide any guarantees. The panel is removed in Container Cloud 2.26.1 (Cluster releases 17.1.1 and 16.1.1). Therefore, consider this section as obsolete for OpenSearch.

Using the following panels in the OpenSearch and Prometheus dashboards, you can view details about the storage usage on managed clusters. These details allow you to calculate the possible retention time based on provisioned storage and its average usage:

  • OpenSearch dashboard:

    • Shards > Estimated Retention

    • Resources > Disk

    • Resources > File System Used Space by Percentage

    • Resources > Stored Indices Disk Usage

    • Resources > Age of Logs

  • Prometheus dashboard:

    • General > Estimated Retention

    • Resources > Storage

    • Resources > Storage by Percentage

To calculate the storage retention time:

  1. Log in to the Grafana web UI. For details, see Access StackLight web UIs.

  2. Assess the OpenSearch and Prometheus dashboards. For details on Grafana dashboards, see View Grafana dashboards.

  3. On each dashboard, select the required period for calculation.

    Tip

    Mirantis recommends analyzing at least one day of data collected in the respective component to benefit from results presented on the Estimated Retention panels.

  4. Assess the Cluster > Estimated Retention panel of each dashboard.

    The panel displays maximum possible retention days while other panels provide details on utilized and available storage.

  5. If persistent volumes of some StackLight components share storage, partition the storage logically to separate components before estimating the retention threshold. This is required since the Estimated Retention panel uses the entire provisioned storage as the calculation base.

    For example, if StackLight is deployed in the default HA mode, then it uses Local Volume Provisioner that provides shared storage unless two separate partitions are configured for each cluster node for exclusive use of Prometheus and OpenSearch.

    Two main storage provisioners are OpenSearch and Prometheus. The level of storage usage by other StackLight components is relatively low. For example, you can share storage logically as follows:

    • 35% for Prometheus

    • 35% for OpenSearch

    • 30% for other components

    In this case, take 35% of the calculated maximum retention value and set it as threshold.

  6. In the Prometheus dashboard, navigate to Resources (Row) > Storage (Panel) > total provisioned disk per pod (Metric) to verify the retention size for the Prometheus storage.

    If both retention time and size are set, Prometheus applies retention to the first reached threshold.

    Caution

    Mirantis does not recommend setting the retention size to 0 and replying on the retention time only.

You can change the retention settings through either the web UI or API:

  • Using the Container Cloud web UI, navigate to the Configure cluster menu and use the StackLight tab

  • Using the Container Cloud API:

    • For OpenSearch, use the logging.retentionTime parameter

    • For Prometheus, use the prometheusServer.retentionTime and prometheusServer.retentionSize parameters

For details, see Change a cluster configuration and Configure StackLight.