Ceph

Ceph monitors use their node host networks to interact with Ceph daemons. Ceph daemons communicate with each other over a specified cluster network and provide endpoints over the public network.

The messenger V2 (msgr2) or earlier V1 (msgr) protocols are used for communication between Ceph daemons.

Ceph daemon

Network

Protocol

Port

Description

Consumers

Manager (mgr)

Cluster network

msgr/msgr2

6800

Listens on the first available port of the 6800-7300 range

csi-rbdplugin,
csi-rbdprovisioner,
rook-ceph-mon

Metadata server (mds)

Cluster network

msgr/msgr2

6800

Listens on the first available port of the 6800-7300 range

csi-cephfsplugin,
csi-cephfsprovisioner

Monitor (mon)

LCM host network

msgr/msgr2

msgr:3300,
msgr2:6789

Monitor has separate ports for msgr and msgr2

Ceph clients
rook-ceph-osd,
rook-ceph-rgw

Ceph OSD (osd)

Cluster network

msgr/msgr2

6800-7300

Binds to the first available port from the 6800-7300 range

rook-ceph-mon,
rook-ceph-mgr,
rook-ceph-mds

Ceph network policies

Available since 2.26.0 (17.1.0 and 16.1.0)

Ceph Controller uses the NetworkPolicy objects for each Ceph daemon. Each NetworkPolicy is applied to a pod with defined labels in the rook-ceph namespace. It only allows the use of the ports specified in the NetworkPolicy spec. Any other port is prohibited.

Ceph daemon

Pod label

Allowed ports

Manager (mgr)

app=rook-ceph-mgr

6800-7300,
9283

Monitor (mon)

app=rook-ceph-mon

3300,
6789

Ceph OSD (osd)

app=rook-ceph-osd

6800-7300

Metadata server (mds)

app=rook-ceph-mds

6800-7300

Ceph Object Storage (rgw)

app=rook-ceph-rgw

Value from spec.cephClusterSpec.objectStorage.rgw.gateway.port,
Value from spec.cephClusterSpec.objectStorage.rgw.gateway.securePort