Univention Bugzilla – Bug 36640
Memory leak in UVMM daemon
Last modified: 2015-04-16 10:49:39 CEST
See the traceback below. The crash of UMC was caused by steadily growing memory consumption of multiple UVMM daemon processes. +++ This bug was initially created as a clone of Bug #36638 +++ The univention-management-console-server is crashed with the following traceback. 13.11.14 12:31:22.304 MAIN ( ERROR ) : Traceback (most recent call last): File "/usr/sbin/univention-management-console-server", line 209, in <module> umc_daemon.do_action() File "/usr/lib/pymodules/python2.7/daemon/runner.py", line 186, in do_action func(self) File "/usr/lib/pymodules/python2.7/daemon/runner.py", line 131, in _start self.app.run() File "/usr/sbin/univention-management-console-server", line 192, in run notifier.loop() File "/usr/lib/pymodules/python2.7/notifier/nf_generic.py", line 284, in loop step() File "/usr/lib/pymodules/python2.7/notifier/nf_generic.py", line 271, in step not __sockets[ cond ][ fd ]( sock_obj ): File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/server.py", line 168, in _receive self._handle(state, msg) File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/server.py", line 286, in _handle state.processor.request(msg) File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/session.py", line 293, in request self.handle_request_command(msg) File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/session.py", line 797, in handle_request_command mod_proc = ModuleProcess(module_name, debug=MODULE_DEBUG_LEVEL, locale=self.i18n.locale) File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/session.py", line 164, in __init__ self.__pid = self.__process.start() File "/usr/lib/pymodules/python2.7/notifier/popen.py", line 135, in start self.pid = subprocess.Popen( cmd, shell = self._shell ).pid File "/usr/lib/python2.7/subprocess.py", line 679, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1153, in _execute_child self.pid = os.fork() OSError: [Errno 12] Nicht gen?gend Hauptspeicher verf?gbar
Do you have any logfiles to analyse this behaviour? UVMMd does require a large amount of memory, especially if EC2 connections are configured. But we did not see a memory-leak in uvmm-daemon itself.
See Bug #35977 comment 3 for a UVMMd memory leak with form info on debugging that kind of problems using "python-meliae".
Reported via feedback: Ticket #2015021821000235
More feedback from the partner: They have three KVM servers which are managed through UVMMd. UVMMd runs on two systems.
Using meliae I've found that xml.dom.minidom seems to have a memory leak: <http://stackoverflow.com/questions/26787026/memory-leak-parsing-xml-using-xml-dom-minidom> r59486 | Bug #36640 UVMM: Fix ucslint errors/warning r59485 | Bug #36640 UVMM: Bump version r59483 | Bug #36640 UVMM: Fix memory leak in XML parsing r59482 | Bug #36640 UVMM: Split/add unit test r59481 | Bug #36640 UVMM: Add memory debugger Package: univention-virtual-machine-manager-daemon Version: 4.0.24-2.592.201503291138 Branch: ucs_4.0-0 Scope: errata4.0-1 r59487 | Bug #36635,Bug #36640 UVMM: YAML 2015-03-29-univention-virtual-machine-manager-daemon.yaml
r59498 | Bug #36640 UVMM: Fix memory leak in XML parsing lxml does not allow setting an attribute to None Package: univention-virtual-machine-manager-daemon Version: 4.0.24-3.593.201503301426 Branch: ucs_4.0-0 Scope: errata4.0-1 r59499 | Bug #36640 UVMM: Fix memory leak in XML parsing YAML 2015-03-29-univention-virtual-machine-manager-daemon.yaml
OK: r59498 | Bug #36640 UVMM: Fix memory leak in XML parsing lxml does not allow setting an attribute to None -> Creating new instances is once more possible Reopen: As discussed, the memory leak is still present, as tested with the most recent version on laiva
# while cat /proc/$(</etc/runit/univention-virtual-machine-manager-daemon/supervise/pid)/statm;do sleep 1;done shows VmRSS to increase permanently in fixed intervals. See <http://xen12.knut.univention.de/collectd/bin/index.cgi?hostname=laiva.knut.univention.de&plugin=processes×pan=86400&action=show_selection&ok_button=OK> for a long-term trend. # gdb -p ... -ex 'generate-core-file' ; strings did show a lot of XML data. # valgrind --leak-check=full --show-reachable=yes --log-file=val.log val.py shows XMLDesc() leaking memory. r14565 | Bug #36640 libvirt: Memory leak Upstream commit 4acfb169400497da2a82a849dc8aaa65f88ac6d1 fixed at least one huge memory leak, but the memory usage of UVMMd is still not stable (= increasing). I will let collectd run some more to get a longer trend as I'm not yet convinced that the problem is fixed now. $ repo_admin.py --cherrypick -r 4.0 --releasedest 4.0 --dest errata4.0-1 -p libvirt-python Package: libvirt-python Version: 1.2.6-2.3.201504081137 Branch: ucs_4.0-0 Scope: errata4.0-1 r59644 | Bug #36640 libvirt-python: Memory leak 2015-04-08-libvirt-python.yaml r59631 | Bug #36640 UVMM: Fix optional keymap in XML parsing Package: univention-virtual-machine-manager-daemon Version: 4.0.24-7.597.201504081138 Branch: ucs_4.0-0 Scope: errata4.0-1
Created attachment 6812 [details] Round-Robin Database: UVMMd Virtual Memory usage UVMMd still shows a memory leak, but a lot less then before. # rrdtool graph ps_vm.png -s '09:00 2015-04-07' -e '09:00 2015-04-09' DEF:value=ps_vm.rrd:value:AVERAGE LINE1:value#0000FF:"PsVm" # display ps_vm.png If it becomes a problem again, clone this bug for further memory leak debugging: * Use Valgrind * Suppress "use-once" allocations <http://valgrind.org/docs/manual/manual-core.html#manual-core.suppress>
Created attachment 6813 [details] Simple test script The program shows an ever increasing memory usage.
(In reply to Philipp Hahn from comment #10) > Created attachment 6813 [details] > Simple test script > > The program shows an ever increasing memory usage. That's wrong: python2.7-dbg has the memory leak - the script is fine, so no easy way to re-produce the leak so far.
OK: YAML r59644 | Bug #36640 libvirt-python: Memory leak OK: r59631 | Bug #36640 UVMM: Fix optional keymap in XML parsing OK: r14565 | Bug #36640 libvirt: Memory leak -> Verified
<http://errata.univention.de/ucs/4.0/147.html> <http://errata.univention.de/ucs/4.0/150.html>