NVLink topology-aware scheduling of multi-GPU instances

TechPreview

Multi-GPU servers, such as NVIDIA HGX systems used for AI training, inference, and similar HPC workloads, often have several (typically four or eight) discrete GPUs put on one host and connected with a high-bandwidth on-board fabric such as NVIDIA NVLink or NVSwitch.

Those workloads exchange large tensors between GPUs and depend on that on-board fabric rather than on PCIe or the network. The fabric is a property of the board and is often built in a way that splits GPUs into more than one domain: two groups of four, NVLink pairs, or separate baseboards. Between domains, traffic crosses PCIe, the CPU, or the datacenter network, which is much slower. A multi-GPU OpenStack instance must therefore receive devices from the same domain. If the scheduler mixes GPUs from different domains, the instance does not benefit from the interconnect.

Vanilla PCI passthrough does not model that interconnect. An operator who tries to encode every valid pair or quad as a distinct PCI alias ends up with a combinatorial flavor set. Cloud users then guess which flavor still has a free GPU tuple, and a failed boot surfaces later as an asynchronous ERROR instance rather than a clear inventory miss.

NVLink topology-aware scheduling is a feature in MOSK that publishes PCI GPUs into the Placement service, enables the operator to encode NVLink grouping as a resource-provider tree, and enables a cloud user to request N GPUs of a given type. The instance scheduler then selects only devices that share one NVLink group. There is no fallback to GPUs from different domains.

This capability is distinct from GPU virtualization (time-sliced vGPU and MIG), which partitions a single physical GPU for several guests. NVLink-aware scheduling applies to PCI passthrough of whole GPUs that remain interconnected.

Prerequisites

OpenStack Caracal, Epoxy, or Gazpacho.
PCI passthrough already viable on the target compute nodes: VT-d or equivalent IOMMU, and device_spec / alias as described in Configure PCI passthrough for guests.
Physical GPUs in PCI-passthrough mode on HGX-class (or similar) hosts whose NVLink or NVSwitch grouping the operator can describe. Do not mix this model with vGPU or MIG on the same cards.
Credentials that can call the Placement CLI or API, so the operator can create group resource providers and re-parent PCI GPUs into an NVLink tree. Rockoon and the OpenStackDeployment custom resource do not perform that reshape.
A static topology: the operator must know which GPUs share an NVLink or NVSwitch domain. MOSK does not discover the topology automatically.

Implementation overview

OpenStack can already publish each passthrough GPU into Placement. With PCI-in-Placement, the compute service reports every matching device as a child of the compute node, with inventory CUSTOM_PCI_<VENDOR>_<PRODUCT>. That first tree is flat, every GPU hangs directly under the compute node.

Upstream OpenStack cannot ask for cards from the same NVLink pair, because it labels each GPU request with a random ID that a flavor cannot name. MOSK enables the flavor to give those requests stable names so a user can request GPUs from the same NVLink pair.

GPU placement - flat

The cloud operator needs to manually reshape the initial tree based on the physical connectivity of the GPUs. Nova does not discover NVLink topology or create the extra nodes. GPUs that share an NVLink domain must sit under the same parent group. Those group nodes carry labels such as CUSTOM_PCI_GROUP2 and CUSTOM_PCI_GROUP4 and have no devices of their own.

Nested groups of two under groups of four match 1 / 2 / 4 GPU instance sizes on an eight-GPU server board. An eight-GPU flavor does not need a group-of-8 provider as there is no invalid way to pick eight of eight GPUs.

compute node (root resource provider)
├── nvlink-group-of-4-a      trait CUSTOM_PCI_GROUP4
│   ├── nvlink-group-of-2-a  trait CUSTOM_PCI_GROUP2
│   │   ├── <node>_pci_<addr0>
│   │   └── <node>_pci_<addr1>
│   └── nvlink-group-of-2-b  trait CUSTOM_PCI_GROUP2
│       ├── <node>_pci_<addr2>
│       └── <node>_pci_<addr3>
└── nvlink-group-of-4-b      trait CUSTOM_PCI_GROUP4
    └── ...

After the reshape, Placement can require named GPU requests to come from the same branch of the tree. That same_subtree constraint means, for example, that two GPUs must sit under the same group-of-2 parent. Nova exposes it as the placement:same_subtree extra spec in instance flavor definition.

GPU placement - grouped

When a cloud user asks for two GPUs, the flavor names those requests _h200-0 and _h200-1 with an optional third field on pci_passthrough:alias (<alias>:2:<prefix>). The flavor lists those names in placement:same_subtree so both cards come from one NVLink pair, and it sets placement:group_policy. If the flavor also requires the group-of-2 label, an option allows extra labels on GPU requests. After the Nova scheduler picks particular cards, the host attaches those same cards, not the first GPUs that look free. The user does not have to pick PCI addresses or a separate flavor per physical GPU pair.

Note

MOSK does not manage Shared NVSwitch virtualization as a supported lifecycle. NVIDIA Fabric Manager partition activation and deactivation are outside of this Technical Preview. See NVIDIA Fabric Manager documentation: Shared NVSwitch virtualization model.

Configuration parameters

Set the following keys in the OpenStackDeployment custom resource. For the procedure, see Configure NVLink topology-aware scheduling of multi-GPU instances.

pci.device_spec: spec:nodes:<NODE-LABEL>::<NODE-LABEL-VALUE>:services:compute:nova:nova_compute:values:conf:nova:pci:device_spec. Selects which physical PCI devices compute nodes may attach.
pci.alias: spec:services:compute:nova:values:conf:nova:pci:alias and spec:nodes:<NODE-LABEL>::<NODE-LABEL-VALUE>:services:compute:nova:nova_compute:values:conf:nova:pci:alias. Names the device type used in flavor extra specs.
pci.report_in_placement: spec:services:compute:nova:values:conf:nova:pci:report_in_placement. Set this cluster-wide together with filter_scheduler.pci_in_placement. The scheduler is shared, so every compute that reports PCI devices must publish them in Placement; each matching PCI device is published as a nested resource provider. Once enabled, do not turn it off while PCI workloads remain. Reversing it is not a supported operation.
filter_scheduler.pci_in_placement: spec:services:compute:nova:values:conf:nova:filter_scheduler:pci_in_placement. When true, PCI requests are satisfied from Placement inventories rather than from the legacy PCI filter alone.
filter_scheduler.allow_pci_traits_in_flavor: spec:services:compute:nova:values:conf:nova:filter_scheduler:allow_pci_traits_in_flavor. Optional. When true, flavors may attach extra required or forbidden traits to the PCI request groups created from pci_passthrough:alias.
Placement resource-provider tree: Group resource providers, parent relationships, and traits such as CUSTOM_PCI_GROUP2. Maintained only through the Placement CLI or API. Trait names must match the flavor extra specs.
Flavor extra specs: pci_passthrough:alias=<alias>:<count>:<group_prefix>, placement:same_subtree, placement:group_policy, and trait_<group>:<TRAIT>=required. The optional third alias field makes Placement group names stable (<group_prefix>-0, <group_prefix>-1, and so on). Prefer placement:group_policy over group_policy, which collides with the aggregate extra-spec filter. These extra specs replace a distinct flavor per physical GPU tuple.

Known limitations

Note

Some of these restrictions may be lifted in future releases of MOSK.

Topology is operator-maintained and static. There is no automatic NVLink or NVSwitch discovery.
NVIDIA Fabric Manager / fmpm / libvirt hooks for NVSwitch partition activation and deactivation are not a MOSK-supported lifecycle.
NUMA affinity and NVLink grouping can conflict on asymmetric GPU-to-NUMA maps. For example, if a system has two sockets, two NUMA nodes each, and eight GPUs split 3-1-3-1 across four NUMA nodes, a flavor for eight GPUs must request exactly four NUMA nodes, a flavor for four GPUs must request at least two NUMA nodes, and a flavor for two GPUs can request one NUMA node but then can use only half of the GPUs, or request two NUMA nodes and potentially get an asymmetric NUMA allocation. If the flavor does not request dedicated CPUs or NUMA, Nova does not apply PCI-NUMA affinity. Workload-level NUMA versus NVLink trade-offs are not characterized in MOSK yet.
Live migration and hot-plug of passthrough GPUs are not claimed beyond the existing PCI passthrough limits.
There is no Horizon or other GUI for GPU topology visualization.