User Guide - libvirt

virtio-mem support in libvirt is also documented in the libvirt domain XML documentation and in the libvirt knowledge base on memory devices.

Important Current Limitations

libvirt limitations are actually due to QEMU limitations. Please also consider relevant incomaptible technologies, espeially:

  • Mlock'ing memory with virtio-mem devices is not supported yet and trying to start the domain will fail.
  • Encrypted/secure virtualization is not supported.
  • Combining virtio-mem with virtio-balloon inflation/deflation is not supported. For example, virsh setmem cannot consider virtio-mem device memory but only initial memory.

Updates

v9.6.0

  • It is no longer required to setup a single vNUMA node just to make <maxMemory\> configuration possible. Libvirt will create a single vNUMA node in that case automatically.

v9.5.0

  • slots no longer have to be specified for the <maxMemory\> setting with virtio-mem. A virtio-mem device consumes no (DIMM) slot and libvirt now properly handles that.

v9.4.0

  • virtio-mem devices support the <address/> attribute to specify the address where the device is placed in guest physical memory. Usually, the address selection is automatically performed by QEMU, but for some migration scenarios, it is required on the migration destination.

v8.1.0

  • Libvirt now supports memory preallocation for virtio-mem devices, and consequently cleanly supports huge pages (hugetlb) and shared memory (shmem). Preallocation can either be enabled manually (e.g., <allocation mode="immediate"/>) or it is enabled implicitly (e.g., with <hugepages>). Note that virtio-mem does not preallocate memory when the domain is started, but when the virtio-mem device actually tries plugging memory to the VM.

Domain XML

With virtio-mem there is an important difference between the domain XML of a running domain and a shutdown domain: especially the currentMemory semantics differ and the current size of virtio-mem devices is only available for a running domain.

maxMemory

The maxMemory size specifies the maximum amount of memory the VM can consume throughout its lifetime, including initial memory and any memory devices. (virtio-mem devices, DIMMs, NVDIMMs, virtio-pmem devices, ...)

Before v9.5.0, it was required to reserve one "slot" for each memory device. Assuming we wanted a maximum VM size of 128 GiB and have two virtio-mem devices:

<maxMemory slots='2' unit='GiB'>128</maxMemory>

memory and currentMemory

The memory and currentMemory sizes will get recalulcated by libvirt automatically and don't have to be set explicitly. In this description we assume that memory ballooning (e.g., virsh setmem) is not used, which can further affect currentMemory but does not apply to virtio-mem device memory.

For a shutdown domain, both correspond to the combined size of initial memory and all defined memory devices. Assuming we have 32 GiB of initial memory and two virtio-mem devices with a maximum size of 48 GiB each:

<memory unit='KiB'>134217728</memory>
<currentMemory unit='KiB'>134217728</currentMemory>

For a running domain, memory corresponds to the combined size of initial memory and all (i.e., coldplugged and hotplugged) memory devices.

For a running domain, currentMemory corresponds to the currently available memory to the VM: in case of virtio-mem means that only the currently plugged device memory is considered. Assuming we have 32 GiB of initial memory and two virtio-mem devices with a maximum size of 48 GiB each that don't expose any memory to the domain yet ("no memory plugged"):

<memory unit='KiB'>134217728</memory>
<currentMemory unit='KiB'>33554432</currentMemory>

memoryBacking

memoryBacking can, but doesn't have to be, specified. For example, to enable shared memory:

<memoryBacking>
  <source type='memfd'/>
  <access mode="shared"/>
</memoryBacking>

Or to enable file-backed memory (which requires a file system that supports sparse files):

<memoryBacking>
    <source type='file'/>
    <access mode='shared'/>
</memoryBacking>

However, not all memoryBacking options are supported with virtio-mem devices yet:

  • hugepages is only supported starting with libvirt v8.1.0 if QEMU supports it.
  • allocation mode="immediate" is only supported starting with libvirt v8.1.0 if QEMU supports it.
  • locked is not supported yet in virtio-mem on lower layers and should not be used.

numa

Before libvirt v9.6.0, the usage of memory devices required a vNUMA configuration for the domain, even if it was just a single vNUMA node. Note that newer libvirt automatically creates a single vNUMA node when required.

The size of initial memory was set via the numa attribute. Assuming we want 16 VCPUs on 2 vNUMA nodes, whit 32 GiB of initial memory each:

  <vcpu placement='static'>16</vcpu>
  ...
  <cpu ...>
  ...
    <numa>
      <cell id='0' cpus='0-7' memory='16' unit='GiB'/>
      <cell id='1' cpus='8-15' memory='16' unit='GiB'/>
    </numa>
  </cpu>

Assuming we want 8 VCPUs on a single NUMA node with 4 GiB:

<vcpu placement='static'>8</vcpu>
...
<cpu ...>
...
  <numa>
    <cell id='0' cpus='0-7' memory='4' unit='GiB'/>
  </numa>
...
</cpu>

NUMA node tuning via numatune is compatible with virtio-mem.

memory model='virtio-mem'

Under devices we specify the actual virtio-mem devices. For example, if we define two virtio-mem devices with a maximum size of 48 GiB each, one device assigned to one of two vNUMA nodes:

<devices>
  ...
  <memory model='virtio-mem'>
    <target>
      <size unit='GiB'>48</size>
      <node>0</node>
      <block unit='MiB'>2</block>
      <requested unit='KiB'>16777216</requested>
      <current unit='KiB'>16777216</current>
    </target>
    <alias name='ua-virtiomem0'/>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
  </memory>
  <memory model='virtio-mem'>
    <target>
      <size unit='GiB'>48</size>
      <node>1</node>
      <block unit='MiB'>2</block>
      <requested unit='KiB'>0</requested>
      <current unit='KiB'>0</current>
    </target>
    <alias name='ua-virtiomem1'/>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
  </memory>
  ...
 </devices>

size

The size under target defines the maximum size of the virtio-mem device. In our example, it's 48 GiB. It has to be multiples of the block size.

node

The node under target defines the assigned vNUMA node for the virtio-mem device.

block

The block size under target defines the device block size of the virtio-mem device. It corresponds the hot(un)plug granularity on the hypervisor side. It has to be a power of two, at least as big as the page size of the memoryBacking (e.g., 1 GiB with gigantic pages on x86-64) and should be at least be the Transparent Huge Page (THP) size (e.g., 2 MiB on x86-64); 2 MiB on x86-64 is usually a good choice.

When vfio/mdev is used, the block size might have to be increased due to limited vfio/mdev mappings: see the QEMU user guide for details.

requested

The requested size under target specifies how much memory we would like the domain to consume via a specific virtio-mem device. It's a request towards the domain, and to which degree the domain is able to fulfill that request is visible via via the current size. Note that there is usually a delay between changing the requested size and observing a change of the current size.

In some cases, the domain might not be able to fulfil the request at all -- especially when running domains without virtio-mem support or when requesting to hotunplug memory without proper preparations inside the VM (e.g., using ZONE_MOVABLE for hotplugged memory under Linux). Also, the domain might not be able to fulfill the request completely in case it has different granularity restrictions -- for example, virtio-mem in older Linux versions supported 4 MiB granularity, and upstream Linux supports 2 MiB granularity; having the requested size not be multiple of 4 MiB consequently cannot be processed completely by an older Linux domain. There are more corner cases.

The requested size has to be multiples of the block size and cannot exceed the size. In general, a domain cannot consume more than the requested size via the virtio-mem device, except when reducing it and the domain cannot fulfill the request (completely). Usually, the domain will retry regularly to eventually fulfill the requests as good as possible -- like retrying to unplug memory until the current size matches the requested size,

Changing the the requested size for a running domain (see virsh update-memory-device) corresponds to a hot(un)plug request. Setting it for a shutdown domain allows for the starting domain to consume memory via the virtio-mem device directly during boot, to eventually hotunplug that memory later.

current

For a running domain, there is an additional current size under target, showing how much memory the virtio-mem device currently provides to the domain ("plugged memory").

The current size changes based on resize requests as the domain tries to fulfill a request; however, it also change due to other events, for example, when rebooting the domain.

alias

The user can specify an alias, to be used in order to change the requested size of a virtio-mem device later (see virsh update-memory-device). User-defined aliases in libvirt have to start with the "ua-" prefix.

address

On x86-64, virtio-mem devices are actually PCI devices. The address is generated by libvirt automatically, but it can be specified manually just like for any other PCI device.

virsh update-memory-device

There is a new virsh command update-memory-device to change the requested size of a virtio-mem device either dynamically for the running domain (--live) or statically for the next boot (--config). Changing the requested size for a running domain corresponds to a hot(un)plug request.

There are various different options to select a virtio-mem device to resize. If the domain only has a single virtio-mem device:

$ virsh update-memory-device $DOMAIN --requested-size $SIZE

If the domain has a single virtio-mem device per NUMA node:

$ virsh update-memory-device $DOMAIN --node $NODE --requested-size $SIZE

If the domain has a mutiple virtio-mem device per NUMA node (or just wanting to use the \):

$ virsh update-memory-device $DOMAIN --alias $ALIAS --requested-size $SIZE

More details are available in the libvirt documentation or via:

$ virsh update-memory-device --help

results matching ""

    No results matching ""