Univention Bugzilla – Bug 32835
Bad UVMMd performance
Last modified: 2023-06-28 10:46:27 CEST
UVMMd shows a constant high load in an environment with 95 hosts and 300 VMs. "uvmm frequency 60000" only improves the situation slightly. "vmstat 1" show 6.000 interrupts / context-switches per second, which goes up to >20.000 on every update interval. 1. On start-up all 95 connections are opened nearly simultaneously, so all connections are polled at nearly the same time. It would probably help to spread the load over time. 2. For each connection a thread is tarted. Looking at "lsof" showed that for each connection one socket and two pipes are opened. This could be lessened by moving to a Worker-model 3. The event notification system of libvirt should be re-evaluated: At least for KVM the regular polling could be (as far as I currently know) disabled. HOSTS=95 VMS=5 # create that many hosts with that many VMs each using KVM tmp=$(TMPDIR=/tmp tempfile -d /tmp -m 0644) ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa for ((h=0;h<$HOSTS;h+=1)) do user="host${h}" useradd -m -p '$1$bkTt4/My$imPMUFg0YHV8W/UeCkF.X0' -G kvm "$user" # ,libvirt #adduser "$user" libvirt #adduser "$user" kvm install -m 0711 -o "$user" -d "/home/host$h/.ssh" install -m 0600 -o "$user" /root/.ssh/id_rsa.pub "/home/$user/.ssh/authorized_keys" for ((vm=0;vm<$VMS;vm+=1)) do cat >"$tmp" <<__XML__ <domain type='kvm'> <name>host${h}vm${vm}</name> <memory unit='MiB'>512</memory> <os> <type arch='x86_64' machine='pc'>hvm</type> <boot dev='hd'/> <boot dev='cdrom'/> </os> <features> <acpi/> <apic/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='cdrom'> <source file='/var/lib/libvirt/images/dummy.iso'/> <target dev='hda' bus='ide'/> <readonly/> </disk> <disk type='file' device='disk'> <source file='/var/lib/libvirt/images/dummy.qcow2'/> <target dev='hda' bus='ide'/> </disk> <interface type='bridge'> <source bridge='br0'/> <model type='e100'/> </interface> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='de'/> <video> <model type='cirrus' vram='9216' heads='1'/> </video> </devices> </domain> __XML__ su -c "virsh -c qemu:///session define '$tmp'" "$user" done done for ((h=0;h<95;h+=1)) do echo "unix_sock_dir=\"/home/$user/.libvirt\"" >"$tmp" su -c "/usr/sbin/libvirtd -f '$tmp' -d" "$user" uvmm add "qemu+ssh://$user@localhost/session?socket=/home/$user/.libvirt/libvirt-sock" done
# No need for doing the testing over ssh, as UVMMd runs as root for ((h=0;h<95;h+=1)) do uvmm add qemu:///session?socket=/home/host$h/.libvirt/libvirt-sock done
See Bug #33458 and the ticket. UVMM is fast enough at least at the beginning of the process. The performance problem rises slowly so that it can't be used anymore after two days. The customer had also an additional script which polls the servers. Is this bug obsolete if Bug #33458 has been fixed?
(In reply to Stefan Gohmann from comment #2) > Is this bug obsolete if Bug #33458 has been fixed? This bug is no longer urgent, but UVMM still has some design problems like polling all VMs to often. This would be fixable for KVM, but not for Xen, as there no event is triggered on domain-edit.
There is a Customer ID set so I set the flag "Enterprise Customer affected".
This issue has been filed against UCS 4.2. UCS 4.2 is out of maintenance and many UCS components have changed in later releases. Thus, this issue is now being closed. If this issue still occurs in newer UCS versions, please use "Clone this bug" or reopen it and update the UCS version. In this case please provide detailed information on how this issue is affecting you.