Configure NVLink topology-aware scheduling of multi-GPU instances

TechPreview

This procedure enables topology-aware scheduling of PCI-passthrough GPUs that share an NVLink or NVSwitch domain. For the description of the feature, the Placement tree model, supported OpenStack releases, and Technical Preview limits, see NVLink topology-aware scheduling of multi-GPU instances.

MOSK does not automatically reshape the Placement resource-provider tree through Rockoon or OpenStackDeployment custom resource. After Nova reports PCI devices, you maintain grouping with the Placement CLI or API.

Enable PCI reporting into Placement

Complete host and OpenStackDeployment PCI passthrough as described in Configure PCI passthrough for guests, including VT-d or equivalent IOMMU, pci.alias, and pci.device_spec for the GPU vendor and product IDs.

Enable PCI-in-Placement in the OpenStackDeployment custom resource. Set both filter_scheduler.pci_in_placement and pci.report_in_placement cluster-wide under spec:services:compute:nova. The scheduler is shared, so every compute that reports PCI devices must publish them in Placement.

Warning

After you enable pci.report_in_placement, do not disable it while instances still use those PCI devices. Reversing PCI-in-Placement is not a supported day-2 operation.

spec:
  services:
    compute:
      nova:
        values:
          conf:
            nova:
              filter_scheduler:
                pci_in_placement: true
              pci:
                report_in_placement: true
  nodes:
    <NODE-LABEL>::<NODE-LABEL-VALUE>:
      services:
        compute:
          nova:
            nova_compute:
              values:
                conf:
                  nova:
                    pci:
                      alias: '{ "vendor_id":"10de", "product_id":"<PRODUCT>", "device_type":"type-PF", "name":"H200" }'
                      device_spec: |
                        [{"vendor_id": "10de", "product_id": "<PRODUCT>"}]

Where:

<PRODUCT> is the GPU product ID as reported by the host, without the 0x prefix
NVIDIA vendor ID is 10de; NVIDIA H200 is commonly 2335
The name in alias must match the first field of pci_passthrough:alias on the flavor

Wait until nova-compute publishes nested resource providers. Each GPU should appear under the compute node with a name of the form <node-name>_pci_<pci-address-with-underscores> and an inventory of one CUSTOM_PCI_<VENDOR>_<PRODUCT> resource. For NVIDIA devices the vendor ID is 10DE.
```
openstack resource provider list --in-tree <COMPUTE-PROVIDER-UUID> -f yaml
openstack resource provider inventory list <PCI-PROVIDER-UUID> -f yaml
```
Obtain <COMPUTE-PROVIDER-UUID> from openstack resource provider list --name <HYPERVISOR-HOSTNAME>.

Reshape the resource-provider tree

Reshape the tree so that GPUs that can share an NVLink domain sit under the same parent group providers. Group providers have traits but no inventory. Nest groups of 2 under groups of 4 when you offer 1 / 2 / 4 GPU flavors. Do not create a group of 8 on an eight-GPU host, because there is no invalid way to pick eight of eight GPUs.

You can apply the same shape with the Placement CLI as below, or with an operator script that reads fmpm -l and nvidia-smi -q from the libvirt pod and re-parents providers (not shipped as an OpenStackDeployment helper). Trait names you set on group providers must match the flavor extra specs.

The nova-compute service does not undo operator-set parent relationships after a restart. Resource tracker continues to report inventory on the PCI child providers.

Create empty group providers as children of the compute node provider and assign traits. Repeat for each group of 2 and 4 that your hardware supports.

openstack resource provider create \
    --parent-provider <COMPUTE-PROVIDER-UUID> \
    <NODE>-nvlink-group-of-2-a

openstack resource provider trait set \
    --trait CUSTOM_PCI_GROUP2 \
    <GROUP-OF-2-UUID>

openstack resource provider create \
    --parent-provider <COMPUTE-PROVIDER-UUID> \
    <NODE>-nvlink-group-of-4-a

openstack resource provider trait set \
    --trait CUSTOM_PCI_GROUP4 \
    <GROUP-OF-4-UUID>

Nest the groups. Set the parent of each group-of-2 provider to the matching group-of-4 provider. Leave each group-of-4 provider as a child of the compute root provider.
```
openstack resource provider set \
    --parent-provider <GROUP-OF-4-UUID> \
    <GROUP-OF-2-UUID>
```
Re-parent each PCI GPU provider under the group-of-2 provider that matches its NVLink pairing. Use the PCI addresses from the host and the <node-name>_pci_<pci-address-with-underscores> names in Placement.
```
openstack resource provider set \
    --parent-provider <GROUP-OF-2-UUID> \
    <PCI-PROVIDER-UUID>
```

Define instance flavors

Create one flavor per GPU count you want to offer, not one flavor per physical GPU tuple. Use the PCI alias with an optional group prefix so Nova creates stable Placement group names (<group_prefix>-0, <group_prefix>-1, and so on), then constrain those groups with placement:same_subtree.

The extra specs below match the MOSK Nova extension pci_passthrough:alias=<alias>:<count>:<group_prefix>. Use placement:group_policy=none rather than group_policy, which conflicts with the aggregate extra-spec filter. Trait names must match the group providers you created when reshaping the tree.

One GPU:

openstack flavor create --ram <RAM> --vcpus <VCPUS> --disk <DISK> gpu-1 \
    --property pci_passthrough:alias='H200:1'

Two GPUs:

openstack flavor create --ram <RAM> --vcpus <VCPUS> --disk <DISK> gpu-2 \
    --property pci_passthrough:alias='H200:2:_h200' \
    --property placement:group_policy=none \
    --property placement:same_subtree='_h200-0,_h200-1,_group2' \
    --property trait_group2:CUSTOM_PCI_GROUP2=required

Four GPUs:

openstack flavor create --ram <RAM> --vcpus <VCPUS> --disk <DISK> gpu-4 \
    --property pci_passthrough:alias='H200:4:_h200' \
    --property placement:group_policy=none \
    --property placement:same_subtree='_h200-0,_h200-1,_h200-2,_h200-3,_group4' \
    --property trait_group4:CUSTOM_PCI_GROUP4=required

Eight GPUs:

On an eight-GPU host, an eight-GPU flavor does not need placement:same_subtree or a group-of-8 trait.

openstack flavor create --ram <RAM> --vcpus <VCPUS> --disk <DISK> gpu-8 \
    --property pci_passthrough:alias='H200:8'

Replace H200 with the alias name from OpenStackDeployment. Publish the flavor names to cloud users as described in Use NVLink topology-aware scheduling for multi-GPU instances.

Note

On hosts with an asymmetric GPU-to-NUMA map, dedicated CPUs or an explicit NUMA extra spec make Nova enforce PCI-NUMA affinity. For example, if a system has two sockets, two NUMA nodes each, and eight GPUs split 3-1-3-1 across four NUMA nodes, a flavor for eight GPUs must request exactly four NUMA nodes, a flavor for four GPUs must request at least two NUMA nodes, and a flavor for two GPUs can request one NUMA node but then can use only half of the GPUs, or request two NUMA nodes and potentially get an asymmetric NUMA allocation. If the flavor does not request dedicated CPUs or NUMA, that affinity is not applied. See NVLink topology-aware scheduling of multi-GPU instances.

Rebalance the tree

Rebalance when hardware grouping changes, when you alter the 2 / 4 / 8 offering, or when you add compute nodes with GPUs.

Drain or stop instances that hold allocations on PCI providers you will move. Re-parenting providers that still have allocations can leave Placement and Nova disagreeing about inventory. If consumers remain after an instance is deleted, see Troubleshoot orphaned resource allocations.
Create or delete group providers and re-parent PCI providers with the same openstack resource provider create, openstack resource provider trait set, and openstack resource provider set --parent-provider commands as in the initial reshape.

Verify the tree and inventories:

openstack resource provider list --in-tree <COMPUTE-PROVIDER-UUID> -f yaml
openstack resource provider show <PCI-PROVIDER-UUID> -f yaml

Boot a test instance with a two-GPU flavor. Confirm the instance reaches ACTIVE and that the allocated PCI providers share one group-of-2 parent. A request that cannot be satisfied in the same subtree fails scheduling (NoValidHost / instance ERROR), including when NUMA affinity and NVLink grouping cannot be met together.