Topologies and use cases

This section describes the compute-node layout used to validate NVIDIA ASAP² Direct on MOSK and the traffic paths that were exercised. Paths that only look plausible from vendor or upstream behavior, but were not part of this lab, are called out separately. Discovered incompatibilities with MOSK Networking service (OpenStack Neutron) features are summarized here and detailed in Limitations.

Validated host OS network layout

Reserve the ASAP² adapter for tenant traffic offload. Do not share that NIC with MOSK life-cycle management, Kubernetes underlay, or storage networks. Mixing infrastructure and tenant traffic on one SmartNIC is not a supported MOSK LCM layout.

The validated compute node used three LACP bonds:

Example three-bond isolation
Bond	Role	Typical networks
LCM bond	Host management, PXE, Kubernetes LCM	LCM VLAN, standard MTU
Infrastructure bond	Storage and cluster infrastructure	Ceph, Kubernetes pod, and external VLANs, floating-IP / external bridge (`br-fip` / `br-ex`)
ASAP² bond	Dedicated ASAP² datapath	VXLAN underlay VLAN, SR-IOV VFs in `switchdev`, provider physnet on the same adapter

Declare the ASAP² bond (<ASAP_BOND_IFACE> in Configuration and activation) and the tenant underlay VLAN (<TENANT_UNDERLAY_IFACE>) in the compute L2Template. Use the same names in OpenStackDeployment.

VF-LAG — Enslave both ports of the same dual-port ASIC into an IEEE 802.3ad (LACP) bond with transmit hash layer3+4. Cross-card bonding of two PCIe NICs is unsupported. Both PFs (<PF0_IFACE> and <PF1_IFACE> in configuration) must be in switchdev before the ASAP² bond comes up. Use a jumbo MTU on the ASAP² bond and the underlay VLAN so encapsulated frames fit. VF-LAG initialization fails if firmware NUM_OF_VFS exceeds 64.

ToR switch — Configure a matching LACP port-channel (typically MLAG) and trunk the underlay VLAN on that channel.

The following diagram illustrates the network topology used for validation:

asap2_direct-Topology

Underlay, overlay, and physical networks

The validated accelerated datapath is a VXLAN tenant overlay.

Instances attach ports with vnic-type=direct and capabilities: ["switchdev"], with port security disabled. The underlay is a tagged VLAN (<TENANT_UNDERLAY_IFACE>) on the ASAP² bond (<ASAP_BOND_IFACE>).

tunnel_interface in OpenStackDeployment must be set to that VLAN netdev so that OVS sends overlay traffic out the ASAP² underlay. After OVS programs the flow, VXLAN encapsulation and decapsulation run in the SmartNIC.

Provider VLANs on the same SmartNIC were also validated: direct SR-IOV switchdev ports on the ASAP² physnet. The eSwitch performs hardware VLAN push and pop. That is not a self-service tenant VLAN. It is a provider network on the ASAP² adapter. A separate infrastructure physnet stays on the non-ASAP² NICs. To learn how to declare that mapping in OpenStackDeployment, refer to Configuration and activation.

Caution

Do not collocate VXLAN tenant overlay networks and VLAN-based self-service tenant networks on the same SmartNIC. Combined OpenStack Nova scheduler and OVS bond IP-assignment limits make that layout unsupported.

If you need VLAN tenants, keep them off the ASAP² NIC. VLAN tenant networks were not a target for hardware offload in this validation in the first place.

Keep the bond in the kernel (Netplan from L2Template). Do not enslave an active VF-LAG bond into br-ex or an SR-IOV bridge from generic OVS startup. Neutron attaches VF representors to br-int. For details, see Configuration and activation.

Validated use cases and traffic paths

The primary accelerated traffic was east-west TCP and UDP between two instances on different ASAP² compute nodes, on one VXLAN overlay, each with a direct switchdev port:

For the throughput numbers, see Performance.
For slow-path versus fast-path behavior, see Architecture.

Validated use cases and traffic paths
Use case	Path	Encapsulation	Offload	Status
East-west overlay	Guest VF on compute node A to guest VF on compute node B	VXLAN over underlay VLAN	Hardware encap/decap after first-packet miss	Validated (line-rate tests)
First packet and ARP/ND	Same overlay; miss to VF representor, then eSwitch	VXLAN	First packet and ARP/ND on host. Matching packets in silicon	Validated
Provider VLAN access	Guest VF to tagged provider segment on the ASAP² physnet	VLAN	Hardware VLAN push and pop	Validated
VF-LAG to ToR	VF egress hashed across both PFs. Ingress from either member	Underlay VLAN on LACP	Hardware VF-LAG	Validated
VirtIO on overlay	`vnic-type=normal` TAP into host OVS on the same VXLAN	VXLAN	Host OVS on the `switchdev` PF, not the direct VF fast path	Attachment validated; not the line-rate test path
Guest virtual router	Appliance with overlay and provider/external `switchdev` ports	VXLAN and/or VLAN	Eligible hops stay in the eSwitch. No Neutron router	Validated as a workaround for north-south traffic

East-west VXLAN overlay

Two instances, two compute nodes, one VXLAN network, both ports vnic-type=direct with capabilities: ["switchdev"] and port security disabled:

The guest transmits on its VF.
On a flow miss, the eSwitch traps the packet to the VF representor. Open vSwitch classifies it and programs TC Flower (VXLAN encap toward the peer underlay IP, or delivery to a local VF).
Subsequent matching packets: guest VF → eSwitch VXLAN encapsulation → underlay VLAN on the VF-LAG bond → ToR → peer compute node → eSwitch VXLAN decapsulation → peer VF. The host CPU is not on that path until the idle timeout (max-idle 30000 ms in the validated setup) expires.

ARP, IPv6 Neighbor Discovery, and the first packet of each new flow stay on the host. Eligible TCP/UDP flows after that run in the eSwitch. IPv6 tenant unicast was not validated explicitly. It should use the same miss-then-hit path.

Provider VLAN

Create a VLAN network on the ASAP² physnet and attach a direct switchdev port the same way. The instance is on the provider segment directly: guest VF → eSwitch VLAN push/pop → PF or VF-LAG uplink.

Note

That path does not use a Neutron logical router or a floating IP. See Configuration and activation.

VF-LAG high availability

The eSwitch distributes VF egress across both bond members and delivers ingress from either wire to the destination VF. The ToR port-channel must match LACP 802.3ad and trunk the underlay VLAN.

VirtIO on the same overlay

Legacy instances with vnic-type=normal can attach to the same VXLAN overlay. Attachment was validated. Traffic uses host Open vSwitch on the switchdev PF. That is not the direct VF hardware path used for line-rate tests.

Guest virtual router or firewall

Neutron routers do not offload. For north-south that must stay accelerated, attach a guest appliance with two switchdev ports: one on the accelerated VXLAN overlay and one on a provider or external network (VLAN or VXLAN) that is also on the ASAP² path. The appliance performs routing, NAT, or stateful policy in the guest. Overlay and provider hops that match hardware flows still run in the eSwitch.

Compute A                         Fabric                    Compute B
Guest A VF --+                                         +-- Guest B VF
             |                                         |
           eSwitch -- VXLAN encap -- underlay VLAN -- eSwitch
             |         on VF-LAG / ToR                 |
Guest appliance (optional): overlay VF + provider VF
Provider VLAN: Guest VF -- eSwitch VLAN push/pop -- uplink

Non-validated use cases and traffic paths

The following topologies and traffic paths are not part of the blueprint. However, some of them may work on other Neutron or NIC layout combinations. Because this validation did not cover them, do not treat them as supported.

Non-validated use cases and traffic paths
Use case	Why it might work	Offload	Status
Floating IP on a `switchdev` port with centralized FIP/SNAT on dedicated gateway nodes (not DVR on the compute node)	NAT is not performed on the compute node that hosts the VF	None. North-south still traverses a Neutron router and NAT	Not validated. Association was not tested outside DVR
IPv6 tenant unicast	Same eSwitch miss-then-hit path as IPv4 after ND	Eligible flows should offload after the first packet	Not validated explicitly. Expected to work
Geneve tenant overlay	NVIDIA / OVS hardware offload literature includes Geneve	Vendor claim	Not validated. Target overlay was VXLAN only
ASAP² on a single PF (no VF-LAG)	VF-LAG is optional; a single uplink can still run `switchdev`	Same eSwitch model on one uplink	Not validated.
Infrastructure and tenant traffic on one SmartNIC	Some vendor reference designs use one high-speed bond with QoS (PFC/ECN), typically on later adapters	Depends on hardware and lossless fabric. Not characterized on ConnectX-6 Lx	Not validated. Not a supported MOSK LCM layout
Hardware IPsec on VXLAN	Documented for ConnectX-6 Dx and later, VXLAN only	Not available on the validated ConnectX-6 Lx	Out of scope. See Limitations
Live migration of a direct ASAP² port, or vDPA	Newer adapters can be tuned for SR-IOV live migration; vDPA keeps virtio in the guest	Not used on ConnectX-6 Lx	Not validated. Future requirement

Centralized floating IP, if association succeeds, still does not gain ASAP² acceleration. For accelerated external access, attach the instance to a provider VLAN or an external VXLAN through a switchdev port, or use a guest appliance as described in Guest virtual router or firewall.