Architecture

On a MOSK compute node, ASAP² Direct runs the Open vSwitch data plane in the SmartNIC rather than solely in the host kernel. The NIC operates in switchdev mode so that Virtual Functions (VFs) attach to instances while the host keeps paired representor interfaces that Open vSwitch uses to program hardware forwarding rules.

The control plane stays familiar: Networking and Compute services (OpenStack Neutron and Nova) allocate networks and ports, and OVS remains the switch control plane. Eligible flows are then executed in the NIC embedded switch (eSwitch), which reduces host CPU cost for steady-state tenant traffic.

Key components and their roles

Components sit in three layers. The SmartNIC hardware forwards matching traffic in silicon. The host operating system runs Open vSwitch, TC Flower, and VF representors. The OpenStack control plane and guest define networks and attach PCI devices.

asap2_direct-Architecture

SmartNIC hardware

The adapter provides the PCIe functions that the host and the guests bind to, and the embedded switch that forwards offloaded traffic between them:

  • Physical Function (PF)

    The PCIe functions for the adapter’s physical ports. They run in switchdev mode, the Linux driver model that offloads the forwarding data plane from the kernel onto switch hardware, and serve as the uplinks toward the physical network.

  • SR-IOV VF-LAG

    Hardware link aggregation across both ports of the same dual-port ASIC. VF-LAG is optional for ASAP² Direct. For link resiliency, Mirantis recommends VF-LAG for production deployments, so this blueprint validated that path. When VF-LAG is in use, the eSwitch distributes VF egress across the bond members and delivers ingress from either wire to the destination VF.

  • Hardware eSwitch

    The NIC forwarding engine (match-action forwarding database). It classifies packets, applies VLAN and header changes, encapsulates and decapsulates VXLAN where programmed, and switches subsequent packets in silicon.

  • Virtual Function (VF)

    A PCIe endpoint sliced from a parent PF and passed to the guest (vnic-type=direct) so instance traffic enters the hardware switch domain by DMA.

Host operating system

The host keeps the switching control plane and the exception path, and translates OVS forwarding decisions into hardware rules:

  • Open vSwitch (OVS)

    Attaches VF representors to the integration bridge, translates OpenFlow decisions into hardware rules, ages idle offloaded flows, and keeps the software path for traffic that cannot be offloaded.

  • VF Representor

    A Linux network interface (netdev) created by the mlx5e_rep driver in switchdev mode and paired one-to-one with a VF. OVS attaches representors as integration bridge ports for the control and exception path: the first packet of a new flow (and other non-offloaded traffic) is trapped from the VF to the representor so OVS can compute and program hardware rules. After offload, matching packets stay in the eSwitch and no longer traverse the representor on the host CPU.

  • Linux TC Flower

    The packet classifier in the Linux traffic control (tc) subsystem. OVS programs hardware offload through TC Flower; the mlx5_core driver then installs the compiled rule in the eSwitch.

OpenStack control plane and cloud workload

OpenStack defines the tenant networks and accelerated ports, and places the VF into the instance that consumes them:

  • Compute service (OpenStack Nova)

    Allocates host CPU resources and, through the libvirt driver, attaches a VF to the guest VM. Nova evaluates the configured PCI device specification to select an available VF on the compute node and passes that PCI device into the QEMU/KVM instance.

  • Networking service (OpenStack Neutron)

    The ML2/OVS plugin and SR-IOV agent manage tenant virtual networks, subnets, and accelerated port objects. Neutron provisions those ports with --vnic-type direct, a binding profile of {"capabilities": ["switchdev"]}, and --disable-port-security, aligned with the offload-capable physical networks (physnets) and bridges.

  • Guest

    The VF appears in the guest OS as a real PCI network device, not a paravirtual virtio-net NIC. The guest must run the NVIDIA/Mellanox driver to send and receive packets on a direct hardware path into the eSwitch.

Datapath behavior

ASAP² Direct uses a miss-then-hit datapath. The guest transmits on the VF into the eSwitch. A hardware-flow hit switches the packet in silicon to a local VF or the PF uplink. A miss traps to the VF representor, Open vSwitch classifies the packet, TC Flower programs the eSwitch, and the first packet is forwarded.

../../../_images/nvidia-asap2-direct-flow.png

First packet of a new flow (slow path):

  1. The guest transmits the packet on its VF.

  2. The NIC eSwitch has no matching hardware flow yet, so it traps the packet to the host through the VF representor.

  3. Open vSwitch on the host classifies the packet and decides the forwarding actions, for example, VXLAN encapsulation toward the underlay, or delivery to another local VF.

  4. OVS programs those actions into the eSwitch through Linux TC Flower.

  5. The first packet is forwarded according to that decision. Subsequent matching packets can use the hardware path.

Subsequent packets of the same flow (fast path):

  1. The guest again transmits on the VF.

  2. The eSwitch matches the installed hardware flow and switches the packet in silicon—between VFs on the same host, or out the PF/bond uplink toward the ToR—without sending the packet through the host CPU or the representor.

  3. Offloaded flows age out after the configured idle timeout (max-idle in the validated setup), after which a new first-packet miss repeats the slow-path setup.

Traffic that cannot be offloaded continues to use the host software path. That includes ARP, traffic through Neutron routers, and floating-IP NAT. Neutron QoS must be disabled globally for the whole cloud, including the qos service plugin and the Open vSwitch agent extension. Leaving policies off individual ports is not enough. See Limitations and Configuration and activation.