Ceph¶

Ceph monitors use their node host networks to interact with Ceph daemons. Ceph daemons communicate with each other over a specified cluster network and provide endpoints over the public network.

The messenger V2 (msgr2) or earlier V1 (msgr) protocols are used for communication between Ceph daemons.

Ceph daemon	Network	Protocol	Port	Description	Consumers
Manager (`mgr`)	Cluster network	msgr/msgr2	6800, 9283	Listens on the first available port of the 6800-7300 range. Uses 9283 port for exporting metrics.	`csi-rbdplugin`, `csi-rbdprovisioner`, `rook-ceph-mon`
Metadata server (`mds`)	Cluster network	msgr/msgr2	6800	Listens on the first available port of the 6800-7300 range	`csi-cephfsplugin`, `csi-cephfsprovisioner`
Monitor (`mon`)	LCM host network	msgr/msgr2	msgr:3300, msgr2:6789	Monitor has separate ports for `msgr` and `msgr2`	Ceph clients `rook-ceph-osd`, `rook-ceph-rgw`
Ceph OSD (`osd`)	Cluster network	msgr/msgr2	6800-7300	Binds to the first available port from the 6800-7300 range	`rook-ceph-mon`, `rook-ceph-mgr`, `rook-ceph-mds`

Ceph network policies¶

Available since MOSK 24.1

Ceph Controller uses the NetworkPolicy objects for each Ceph daemon. Each NetworkPolicy is applied to a pod with defined labels in the rook-ceph namespace. It only allows the use of the ports specified in the NetworkPolicy spec. Any other port is prohibited.

Ceph daemon	Pod label	Allowed ports
Manager (`mgr`)	`app=rook-ceph-mgr`	6800-7300, 9283
Monitor (`mon`)	`app=rook-ceph-mon`	3300, 6789
Ceph OSD (`osd`)	`app=rook-ceph-osd`	6800-7300
Metadata server (`mds`)	`app=rook-ceph-mds`	6800-7300
Ceph Object Storage (`rgw`)	`app=rook-ceph-rgw`	Value from `spec.cephClusterSpec.objectStorage.rgw.gateway.port`, Value from `spec.cephClusterSpec.objectStorage.rgw.gateway.securePort`