Reference Application for workload monitoring |
2.28.0 |
2.28.2 |
2.28.3 |
Deprecated support for Reference Application on non-MOSK
managed clusters. Due to this deprecation, if the RefAppDown alert
is firing in the cluster, disable refapp.enabled to prevent
unnecessary alerts.
|
StackLight resourcesPerClusterSize parameter |
2.28.0 |
|
|
Deprecated the resourcesPerClusterSize parameter in StackLight configuration
for the sake of the resources parameter. |
Ubuntu Focal for baremetal-based clusters |
2.28.0 |
2.28.0 |
2.29.0 |
Deprecated support for the Focal Fossa Ubuntu distribution in the
baremetal-based clusters for the sake of Jammy Jellyfish.
Warning
During the course of the Container Cloud 2.28.x series, Mirantis
highly recommends upgrading an operating system on your cluster machines to
Ubuntu 22.04 before the next major Cluster release becomes available.
It is not mandatory to upgrade all machines at once. You can upgrade them
one by one or in small batches, for example, if the maintenance window is
limited in time.
Otherwise, the Cluster release update of the Ubuntu 20.04-based clusters
will become impossible as of the Cluster releases introduced in Container
Cloud 2.29.0 (Cluster releases 17.4.0 and 16.4.0) with Ubuntu 22.04 as
the only supported version.
|
VMware vSphere-based clusters |
2.27.2 |
2.27.2 |
2.27.3 |
Suspended support for VMware vSphere-based deployments, including
attachment of existing Mirantis Kubernetes Engine (MKE) clusters that
were originally not deployed by Container Cloud.
Simultaneously, ceased performing functional integration testing of the
vSphere provider and removed the possibility to update a vSphere-based cluster to
Container Cloud 2.27.3 (Cluster release 16.2.3).
Therefore, the final supported version for this cloud provider is
Container Cloud 2.27.2 (Cluster release 16.2.2). If you still require
the feature, contact Mirantis support for further information.
|
Ceph metrics |
2.27.0 |
2.27.0 |
2.28.0 |
Deprecated the performance metric exporter that is integrated into the
Ceph Manager daemon for the sake of the dedicated Ceph Exporter daemon.
Names of metrics will not be changed, no metrics will be removed.
All Ceph metrics to be collected by the Ceph Exporter daemon will change
their labels job and instance due to scraping metrics from
new Ceph Exporter daemon instead of the performance metric exporter of
Ceph Manager:
Values of the job labels will be changed from rook-ceph-mgr to
prometheus-rook-exporter for all Ceph metrics moved to Ceph
Exporter. The full list of moved metrics is presented below. Values of the instance labels will be changed from the metric endpoint
of Ceph Manager with port 9283 to the metric endpoint of Ceph Exporter
with port 9926 for all Ceph metrics moved to Ceph Exporter. The full
list of moved metrics is presented below. Values of the instance_id labels of Ceph metrics from the RADOS
Gateway (RGW) daemons will be changed from the daemon GID to the daemon
subname. For example, instead of instance_id="<RGW_PROCESS_GID>",
the instance_id="a" (ceph_rgw_qlen{instance_id="a"}) will be
used. The list of moved Ceph RGW metrics is presented below.
Therefore, if Ceph metrics to be collected by the Ceph Exporter daemon
are used in any customizations, for example, custom alerts, Grafana
dashboards, or queries in custom tools, update your customizations
to use new labels since Container Cloud 2.28.0 (Cluster releases 16.3.0
and 17.3.0).
ceph_bluefs_.*
ceph_bluestore_.*
ceph_mds_cache_.*
ceph_mds_caps
ceph_mds_ceph_.*
ceph_mds_dir_.*
ceph_mds_exported_inodes
ceph_mds_forward
ceph_mds_handle_.*
ceph_mds_imported_inodes
ceph_mds_inodes.*
ceph_mds_load_cent
ceph_mds_log_.*
ceph_mds_mem_.*
ceph_mds_openino_dir_fetch
ceph_mds_process_request_cap_release
ceph_mds_reply_.*
ceph_mds_request
ceph_mds_root_.*
ceph_mds_server_.*
ceph_mds_sessions_.*
ceph_mds_slow_reply
ceph_mds_subtrees
ceph_mon_election_.*
ceph_mon_num_.*
ceph_mon_session_.*
ceph_objecter_.*
ceph_osd_numpg.*
ceph_osd_op.*
ceph_osd_recovery_.*
ceph_osd_stat_.*
ceph_paxos.*
ceph_prioritycache.*
ceph_purge.*
ceph_rgw_cache_.*
ceph_rgw_failed_req
ceph_rgw_gc_retire_object
ceph_rgw_get.*
ceph_rgw_keystone_.*
ceph_rgw_lc_.*
ceph_rgw_lua_.*
ceph_rgw_pubsub_.*
ceph_rgw_put.*
ceph_rgw_qactive
ceph_rgw_qlen
ceph_rgw_req
ceph_rocksdb_.*
|
SubnetPool resource management
|
2.27.0 |
2.28.0 |
2.29.0 |
Deprecated the SubnetPool resource along with automated subnet creation
using SubnetPool.
Existing configurations that use the SubnetPool object in L2Template
will be automatically migrated to Subnet objects during cluster update
to the Cluster release 17.2.0 or 16.2.0. As a result of migration, existing
Subnet objects will be referenced in L2Template objects instead of
SubnetPool.
Since Container Cloud 2.29.0 (Cluster release 16.4.0), the SubnetPool
object creation will be blocked by Admission Controller.
If you still require this feature, contact Mirantis support for further information.
|
MetalLBConfigTemplate resource management
|
2.27.0 |
2.28.0 |
2.29.0 |
Deprecated the MetalLBConfigTemplate resource. Use the MetalLBConfig
resource instead.
Existing MetalLBConfigTemplate objects and related Subnet objects
are automatically migrated to MetallbConfig during cluster update to
the Cluster release 17.2.0 or 16.2.0.
Since Container Cloud 2.29.0 (Cluster release 16.4.0), the MetalLBConfigTemplate
object creation will be blocked by Admission Controller.
|
maxWorkerUpgradeCount parameter
|
2.27.0 |
|
|
Deprecated the maxWorkerUpgradeCount parameter of the Cluster object.
Use the concurrentUpdates parameter in the UpdateGroup object instead. |
byName in BareMetalHostProfile
|
2.26.0 |
2.26.0 |
2.27.0 |
Deprecated the byName field in the BareMetalHostProfile object.
As a replacement, use a more specific selector, such as byPath,
serialNumber, or wwn. For details, see
BareMetalHostProfile. |
minSizeGiB and maxSizeGiB in BareMetalHostProfile
|
2.26.0 |
|
|
Deprecated the minSizeGiB and maxSizeGiB fields in the
BareMetalHostProfile object.
Instead of floats that define sizes in GiB for *GiB fields, use
the <sizeNumber>Gi text notation such as Ki, Mi, and
so on.
All newly created profiles are automatically migrated to the Gi
syntax. In existing profiles, migrate the syntax manually.
|
The wipe field in spec of BareMetalHostProfile |
2.26.0 |
|
|
Deprecated the wipe field from the spec:devices section of the
BareMetalHostProfile object for the sake of wipeDevice.
For backward compatibility, any existing wipe: true option
is automatically converted to the following structure:
wipeDevice:
eraseMetadata:
enabled: True
For new machines, use the wipeDevice structure in the
BareMetalHostProfile object.
|
Several StackLight logging parameters |
2.25.0 |
2.25.0 |
2.26.0 |
Removed the following logging-related StackLight parameters in the
light of the logging pipeline refactoring:
logging.level (also removed from the logging.externalOutputs
configuration)
logging.syslog.level
logging.retentionTime
elasticsearch.logstashRetentionTime
elasticsearch.retentionTime
|
Regional clusters |
2.25.0 |
2.25.0 |
2.26.0 |
Suspended support for regional clusters of the same or different cloud
provider type on a single management cluster.
Additionally, suspended support for several regions on a single
management cluster. Simultaneously, ceased performing functional
integration testing of the feature and removed the related code in
Container Cloud 2.26.0. If you still require this feature,
contact Mirantis support for further information. |
CentOS and RHEL 7.9 support for the vSphere provider |
2.25.0 |
2.25.0 |
2.26.0 |
Suspended support for CentOS and ended support for the deprecated
RHEL 7.9 operating system by the vSphere provider. Instead, use
RHEL 8.7 or Ubuntu 20.04.
|
Bootstrap v1 |
2.25.0 |
2.25.0 |
2.26.0 |
Deprecated the bootstrap procedure using Bootstrap v1 in the sake of
Bootstrap v2. For details, see Deploy Container Cloud using
Bootstrap v2. |
Default L2Template for a namespace |
2.25.0 |
|
|
Disabled creation of the default L2 template for a namespace.
On existing clusters, clusterRef: default is removed during the
migration process. Subsequently, this parameter is not substituted with
the cluster.sigs.k8s.io/cluster-name label, ensuring the application
of the L2 template across the entire Kubernetes namespace. Therefore,
you can continue using existing default namespaced L2 templates.
|
L2Template clusterRef parameter |
2.25.0 |
|
|
Deprecated the clusterRef parameter located in the L2Template
spec. Use the cluster.sigs.k8s.io/cluster-name label instead.
On existing clusters, this parameter is automatically migrated to the
cluster.sigs.k8s.io/cluster-name label since 2.25.0.
|
L2Template without l3Layout parameters section |
2.9.0 |
2.23.5 |
2.24.0 |
Deprecated the use of the L2Template object without the l3Layout
section in spec. The use of the l3Layout section is mandatory
since Container Cloud 2.24.0.
On existing clusters, the l3Layout section is not added automatically.
Therefore, if you do not have the l3Layout section in L2 templates
of your existing clusters, manually add it and define all subnets
that are used in the npTemplate section. For details, see
Update L2 templates on existing bare metal clusters and Create L2 templates.
Caution
Partial definition of subnets is prohibited.
|
Bare metal dnsmasq.dhcp_range parameter |
2.21.0 |
2.23.0 |
2.24.0 |
Deprecated the dnsmasq.dhcp_range parameter of the
baremetal-operator Helm chart values in the Cluster spec.
Use the Subnet object configuration for this purpose instead.
Since 2.24.0, admission-controller does not accept any changes to
dnsmasq.dhcp_range except removal. Therefore, manually remove this
parameter from the baremetal-operator release spec section
of the Cluster object as described in Migration of DHCP configuration for existing management clusters.
|
MetalLB configInline parameter |
2.24.0 |
2.24.0 |
2.25.0 |
Deprecated the configInline parameter in the metallb Helm chart
values of the Cluster spec. Use the following objects instead
of this parameter:
|
cephDeviceMapping status fields
|
2.24.0 |
2.24.0 |
2.24.0 |
Removed cephDeviceMapping from the status.fullClusterInfo.cephDetails
section of the KaaSCephCluster object because its large size can
potentially exceed the Kubernetes 1.5 MB quota. |
Attachment of MKE clusters |
2.24.0 |
2.24.0 |
2.24.0 |
Suspended support for attachment of existing Mirantis Kubernetes Engine
(MKE) clusters that were originally not deployed by Container Cloud.
Also suspended support for all related features, such as sharing a Ceph
cluster with an attached MKE cluster.
Note
The feature support is renewed since Container Cloud 2.25.2
for vSphere-based clusters.
|
Public cloud providers |
2.23.0 |
2.23.0 |
2.23.0 |
Removed the following cloud providers from the list of supported
Container Cloud providers: AWS, Azure, Equinix Metal with private or
public networking. |
logging.syslog parameter for StackLight
|
2.23.0 |
|
|
Deprecated logging.syslog in favor of logging.externalOutputs
that contains a wider range of configuration options. |
IPaddr status fields
|
2.23.0 |
2.24.0 |
2.25.0 |
|
L2Template status fields
|
2.23.0 |
2.24.0 |
2.25.0 |
|
Subnet and SubnetPool status fields
|
2.23.0 |
2.24.0 |
2.25.0 |
Subnet.Status: labelSetChecksum, statusMessage in
favor of state and messages
SubnetPool.Status: statusMessage in favor of state and
messages
|
IpamHost status fields renaming
|
2.21.0 |
2.21.0 |
2.22.0 |
The following fields of the IpamHost status are renamed
in Container Cloud 2.22.0:
netconfigV2 to netconfigCandidate
netconfigV2state to netconfigCandidateState
netconfigFilesState to netconfigFilesStates (per file)
The format of netconfigFilesState is changed after renaming. The
netconfigFilesStates field contains a dictionary of statuses of network
configuration files stored in netconfigFiles. The dictionary contains
the keys that are file paths and values that have the same meaning for each
file that netconfigFilesState had:
For a successfully rendered configuration file:
OK: <timestamp> <sha256-hash-of-rendered-file>, where a timestamp
is in the RFC 3339 format. For a failed rendering: ERR: <error-message>.
|
Ceph mgr.modules parameter |
2.20.0 |
|
|
Deprecated the mgr.modules parameter in favor of mgr.mgrModules. |
Ceph RADOS Gateway users parameter |
2.20.0 |
|
|
Deprecated the Ceph RADOS Gateway users parameter in favor of
objectUsers. |
Ceph on management and regional clusters |
2.19.0 |
2.19.0 |
2.20.0 |
Removed Ceph cluster deployment from the management and regional
clusters based on bare metal and Equinix Metal with private networking
to reduce resource consumption. Ceph is automatically removed during
the Cluster release update to 11.4.0 or 7.10.0. |
Ceph dataPool field in CephFS specification |
2.19.0 |
|
|
Deprecated the dataPool field in CephFS specification in favor of
dataPools. |
status.l2RenderResult field of the IpamHost object
|
2.19.0 |
|
|
Deprecated the status.l2RenderResult field of the IpamHost
object in the sake of status.NetconfigCandidateState. |
Public networking mode on the Equinix Metal based deployments |
2.18.0 |
|
|
Deprecated the public networking mode on the Equinix Metal based
deployments for the sake of the private network mode. |
Services and parameters related to OpenSearch and Kibana |
2.18.0 |
|
|
Deprecated elasticsearch-master in favor of opensearch-master.
In future releases, the following parameters of the
stacklight.values section will be deprecated and finally replaced
by:
elasticsearch in favor of logging
elasticsearch.retentionTime in favor of logging.retentionTime
resourcesPerClusterSize.elasticsearch in favor of
resourcesPerClusterSize.opensearch
resourcesPerClusterSize.fluentdElasticsearch in favor of
resourcesPerClusterSize.fluentdLogs
resources.fluentdElasticsearch in favor of
resources.fluentdLogs
resources.elasticsearch in favor of resources.opensearch
resources.iamProxyKibana in favor of
resources.iamProxyOpenSearchDashboards
resources.kibana in favor of resources.opensearchDashboards
nodeSelector.component.elasticsearch in favor of
nodeSelector.component.opensearch
nodeSelector.component.fluentdElasticsearch in favor of
nodeSelector.component.fluentdLogs
nodeSelector.component.kibana in favor of
nodeSelector.component.opensearchDashboards
tolerations.component.elasticsearch in favor of
tolerations.component.opensearch
tolerations.component.fluentdElasticsearch in favor of
tolerations.component.fluentdLogs
tolerations.component.kibana in favor of
tolerations.component.opensearchDashboards
stacklightLogLevels.component.fluentdElasticsearch in favor of
stacklightLogLevels.component.fluentdLogs
stacklightLogLevels.component.elasticsearch in favor of
stacklightLogLevels.component.opensearch
stacklightLogLevels.component.kibana in favor of
stacklightLogLevels.component.opensearchDashboards
|
Elasticsearch and Kibana |
2.16.0 |
2.16.0 |
2.18.0 |
Elasticsearch is replaced with OpenSearch, Kibana is replaced with
OpenSearch Dashboards. For details, see Elasticsearch switch to OpenSearch. |
Retention Time StackLight parameter in the Container Cloud
web UI |
2.17.0 |
2.17.0 |
2.17.0 |
Replaced with the Logstash Retention Time,
Events Retention Time, and
Notifications Retention Time parameters. |
logstashRetentionTime for Elasticsearch
|
2.16.0 |
2.25.0 |
2.26.0 |
Deprecated the elasticsearch.logstashRetentionTime parameter in
favor of the elasticsearch.retentionTime.logstash,
elasticsearch.retentionTime.events, and
elasticsearch.retentionTime.notifications parameters. |
iam-api and IAM CLI
|
2.15.0 |
2.17.0 |
2.18.0 |
Deprecated the iam-api service and IAM CLI (the iamctl command).
The logic of the iam-api service required for Container Cloud
is moved to scope-controller. |
manageOsds
|
2.14.0 |
2.16.0 |
2.17.0 |
Deprecated manageOsds in the KaaSCephCluster CR. To remove Ceph
OSDs, see Remove Ceph OSD manually or Automated Ceph LCM
Available since 2.15.0. |
status.nicMACmap
|
2.14.0 |
2.14.0 |
2.15.0 |
Removed nicMACmap in the IpamHost status. Instead, use
the serviceMap field that contains actual information about
services, IP addresses, and addresses interfaces. |
ipam/DefaultSubnet label
|
2.14.0 |
|
|
Deprecated the ipam/DefaultSubnet label of the metadata field of
the Subnet object. |
DISABLE_OIDC
|
2.13.0 |
2.14.0 |
2.14.0 |
Removed the DISABLE_OIDC flag required to be set for custom TLS
Keycloak and web UI certificates during a management cluster deployment.
Do not set this parameter anymore in bootstrap.env. To use your
own TLS certificates for Keycloak, refer to Configure TLS certificates for cluster applications. |
IpamHost status fields:
ipAllocationResult
osMetadataNetwork
|
2.12.0 |
2.21.0 |
2.22.0 |
ipAllocationResult is replaced with State and an
error message, if any, in the Messages list.
osMetadataNetwork is replaced with NetconfigCandidate.
|
maintenance flag
|
2.12.0 |
2.13.0 |
2.14.0 |
Deprecated the Ceph maintenance flag. |
match and match_re keys for notifications routes configuration
|
2.12.0 |
2.13.0 |
2.14.0 |
Replaced with the matchers key. |
IPAM API resources:
created
lastUpdated
versionIpam
|
2.11.0 |
2.11.0 |
2.12.0 |
These resources will be eventually replaced with:
objCreated
objUpdated
objStatusUpdated
|
KEYCLOAK_FLOATING_IP
IAM_FLOATING_IP
|
2.9.0 |
2.9.0 |
2.10.0 |
These IPs are now automatically generated in the MetalLB range
for certificates creation in the VMware vSphere cloud provider. |
Netchecker |
2.9.0 |
2.10.0 |
2.11.0 |
Deprecated the Netchecker service. |
bootUEFI
|
2.8.0 |
2.8.0 |
2.9.0 |
The bootUEFI field in the BareMetalHost configuration is
replaced with bootMode that has the following values:
|
SSH-enabled user names |
2.7.0 |
2.8.0 |
2.9.0 |
All existing SSH user names, such as ubuntu, cloud-user for
vSphere-based clusters, are replaced with the universal mcc-user
that is applicable to any Container Cloud provider and node type,
including Bastion. Starting from Container Cloud 2.9.0,
SSH keys are managed only for mcc-user. |
instanceState, instanceID for AWS machines live statuses
|
2.8.0 |
|
|
Replaced with providerInstanceState that includes the AWS VM ID,
state, and readiness |