Disable TX offloading on NICs used by vRouter¶
In the following cases, a TCP-based service may not work on VMs:
If the setup has nested VMs.
If VMs are running in the ESXi hypervisor.
If the Network Interface Cards (NICs) do not support the IP checksum calculation and generate an incorrect checksum. For example, the Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe NIC cards.
To resolve the issue, disable the transmit (TX) offloading on all OpenStack compute nodes for the affected NIC used by the vRouter as described below.
To identify the issue:
Verify whether ping is working between VMs on different hypervisor hosts and the TCP services are working.
Run the following command for the vRouter Agent and verify whether the output includes the number of
Checksum errors
:kubectl -n tf exec tf-vrouter-agent-XXXXX -c agent -- dropstats
Run the following command and verify if the output includes the
cksum incorrect
entries:kubectl -n tf exec tf-vrouter-agent-XXXXX -c agent -- tcpdump -i <tunnel interface> -v -nn | grep -i incorrect
Example of system response:
tcpdump: listening on <tunnel interface>, link-type EN10MB (Ethernet), capture size 262144 bytes <src ip.port> > <dst ip.port>: Flags [S.], cksum 0x43bf (incorrect -> 0xb8dc), \ seq 1901889431, ack 1081063811, win 28960, options [mss 1420,sackOK,\ TS val 456361578 ecr 41455995,nop,wscale 7], length 0 <src ip.port> > <dst ip.port>: Flags [S.], cksum 0x43bf (incorrect -> 0xb8dc), \ seq 1901889183, ack 1081063811, win 28960, options [mss 1420,sackOK,\ TS val 456361826 ecr 41455995,nop,wscale 7], length 0 <src ip.port> > <dst ip.port>: Flags [S.], cksum 0x43bf (incorrect -> 0xb8dc), \ seq 1901888933, ack 1081063811, win 28960, options [mss 1420,sackOK,\ TS val 456362076 ecr 41455995,nop,wscale 7], length 0
Run the following command for the vRouter Agent container and verify whether the output includes the information about a drop for an unknown reason:
kubectl -n tf exec tf-vrouter-agent-XXXXX -c agent -- flow -l
To disable the TX offloading on NICs used by vRouter:
Open the
TFOperator
custom resource (CR) for editing:kubectl -n tf edit tfoperators.operator.tf.mirantis.com openstack-tf
Specify the
DISABLE_TX_OFFLOAD
variable with the"YES"
value for the vRouter Agent container:spec: controllers: tf-vrouter: agent: containers: - name: agent env: - name: DISABLE_TX_OFFLOAD value: "YES"
Warning
Once you modify the
TFOperator
CR, thetf-vrouter-agent-<XXXXX>
pods will not restart automatically because they use the OnDelete update strategy. Restart such pods manually, considering that the vRouter pods restart causes network services interruption for the VMs hosted on the affected nodes.To disable TX offloading on a specific subset of nodes, use custom vRouter settings. For details, see Custom vRouter settings.
Warning
Once you add a new
CustomSpec
, a new daemon set will be generated and thetf-vrouter-agent-<XXXXX>
pods will be automatically restarted. The vRouter pods restart causes network services interruption for VMs hosted on the affected node. Therefore, plan this procedure accordingly.