Bug 32835 - Bad UVMMd performance
Bad UVMMd performance
Status: CLOSED WONTFIX
Product: UCS
Classification: Unclassified
Component: Virtualization - UVMM
UCS 4.2
amd64 Linux
: P3 normal (vote)
: ---
Assigned To: UCS maintainers
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-10 09:53 CEST by Philipp Hahn
Modified: 2023-06-28 10:46 CEST (History)
1 user (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 4: Minor Usability: Impairs usability in secondary scenarios
Who will be affected by this bug?: 2: Will only affect a few installed domains
How will those affected feel about the bug?: 2: A Pain – users won’t like this once they notice it
User Pain: 0.091
Enterprise Customer affected?: Yes
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional): Large environments
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hahn univentionstaff 2013-10-10 09:53:42 CEST
UVMMd shows a constant high load in an environment with 95 hosts and 300 VMs.
"uvmm frequency 60000" only improves the situation slightly.
"vmstat 1" show 6.000 interrupts / context-switches per second, which goes up to >20.000 on every update interval.

1. On start-up all 95 connections are opened nearly simultaneously, so all connections are polled at nearly the same time. It would probably help to spread the load over time.

2. For each connection a thread is tarted. Looking at "lsof" showed that for each connection one socket and two pipes are opened. This could be lessened by moving to a Worker-model

3. The event notification system of libvirt should be re-evaluated: At least for KVM the regular polling could be (as far as I currently know) disabled.

HOSTS=95 VMS=5 # create that many hosts with that many VMs each using KVM
tmp=$(TMPDIR=/tmp tempfile -d /tmp -m 0644)
ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa
for ((h=0;h<$HOSTS;h+=1))
do
  user="host${h}"
  useradd -m -p '$1$bkTt4/My$imPMUFg0YHV8W/UeCkF.X0' -G kvm "$user" # ,libvirt
  #adduser "$user" libvirt
  #adduser "$user" kvm
  install -m 0711 -o "$user" -d "/home/host$h/.ssh"
  install -m 0600 -o "$user" /root/.ssh/id_rsa.pub "/home/$user/.ssh/authorized_keys"
  for ((vm=0;vm<$VMS;vm+=1))
  do
    cat >"$tmp" <<__XML__
<domain type='kvm'>
  <name>host${h}vm${vm}</name>
  <memory unit='MiB'>512</memory>
  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <boot dev='hd'/>
    <boot dev='cdrom'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='cdrom'>
      <source file='/var/lib/libvirt/images/dummy.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
    </disk>
    <disk type='file' device='disk'>
      <source file='/var/lib/libvirt/images/dummy.qcow2'/>
      <target dev='hda' bus='ide'/>
    </disk>
    <interface type='bridge'>
      <source bridge='br0'/>
      <model type='e100'/>
    </interface>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='de'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
    </video>
  </devices>
</domain>
__XML__
    su -c "virsh -c qemu:///session define '$tmp'" "$user"
  done
done
for ((h=0;h<95;h+=1))
do
  echo "unix_sock_dir=\"/home/$user/.libvirt\"" >"$tmp"
  su -c "/usr/sbin/libvirtd -f '$tmp' -d" "$user"
  uvmm add "qemu+ssh://$user@localhost/session?socket=/home/$user/.libvirt/libvirt-sock"
done
Comment 1 Philipp Hahn univentionstaff 2013-10-10 11:34:13 CEST
# No need for doing the testing over ssh, as UVMMd runs as root
for ((h=0;h<95;h+=1))
do
  uvmm add qemu:///session?socket=/home/host$h/.libvirt/libvirt-sock
done
Comment 2 Stefan Gohmann univentionstaff 2013-11-22 07:47:59 CET
See Bug #33458 and the ticket. UVMM is fast enough at least at the beginning of the process. The performance problem rises slowly so that it can't be used anymore after two days.

The customer had also an additional script which polls the servers.

Is this bug obsolete if Bug #33458 has been fixed?
Comment 3 Philipp Hahn univentionstaff 2014-03-12 14:47:19 CET
(In reply to Stefan Gohmann from comment #2)
> Is this bug obsolete if Bug #33458 has been fixed?

This bug is no longer urgent, but UVMM still has some design problems like polling all VMs to often. This would be fixable for KVM, but not for Xen, as there no event is triggered on domain-edit.
Comment 4 Florian Best univentionstaff 2017-06-28 14:53:02 CEST
There is a Customer ID set so I set the flag "Enterprise Customer affected".
Comment 5 Ingo Steuwer univentionstaff 2020-07-03 20:55:57 CEST
This issue has been filed against UCS 4.2.

UCS 4.2 is out of maintenance and many UCS components have changed in later releases. Thus, this issue is now being closed.

If this issue still occurs in newer UCS versions, please use "Clone this bug" or reopen it and update the UCS version. In this case please provide detailed information on how this issue is affecting you.