Bug 36640 - Memory leak in UVMM daemon
Memory leak in UVMM daemon
Product: UCS
Classification: Unclassified
Component: Virtualization - UVMM
UCS 4.0
Other Linux
: P5 normal (vote)
: UCS 4.0-1-errata
Assigned To: Philipp Hahn
Erik Damrose
Depends on:
  Show dependency treegraph
Reported: 2014-11-13 13:27 CET by Alexander Kläser
Modified: 2015-04-16 10:49 CEST (History)
6 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional): Troubleshooting
Max CVSS v3 score:

Round-Robin Database: UVMMd Virtual Memory usage (145.16 KB, application/octet-stream)
2015-04-09 09:16 CEST, Philipp Hahn
Simple test script (1.98 KB, text/plain)
2015-04-09 09:17 CEST, Philipp Hahn

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Kläser univentionstaff 2014-11-13 13:27:05 CET
See the traceback below. The crash of UMC was caused by steadily growing memory consumption of multiple UVMM daemon processes.

+++ This bug was initially created as a clone of Bug #36638 +++

The univention-management-console-server is crashed with the following traceback. 

13.11.14 12:31:22.304  MAIN        ( ERROR   ) : Traceback (most recent call last):
  File "/usr/sbin/univention-management-console-server", line 209, in <module>
  File "/usr/lib/pymodules/python2.7/daemon/runner.py", line 186, in do_action
  File "/usr/lib/pymodules/python2.7/daemon/runner.py", line 131, in _start
  File "/usr/sbin/univention-management-console-server", line 192, in run
  File "/usr/lib/pymodules/python2.7/notifier/nf_generic.py", line 284, in loop
  File "/usr/lib/pymodules/python2.7/notifier/nf_generic.py", line 271, in step
    not __sockets[ cond ][ fd ]( sock_obj ):
  File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/server.py", line 168, in _receive
    self._handle(state, msg)
  File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/server.py", line 286, in _handle
  File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/session.py", line 293, in request
  File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/session.py", line 797, in handle_request_command
    mod_proc = ModuleProcess(module_name, debug=MODULE_DEBUG_LEVEL, locale=self.i18n.locale)
  File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/session.py", line 164, in __init__
    self.__pid = self.__process.start()
  File "/usr/lib/pymodules/python2.7/notifier/popen.py", line 135, in start
    self.pid =  subprocess.Popen( cmd, shell = self._shell ).pid
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1153, in _execute_child
    self.pid = os.fork()
OSError: [Errno 12] Nicht gen?gend Hauptspeicher verf?gbar
Comment 1 Erik Damrose univentionstaff 2014-11-13 13:33:17 CET
Do you have any logfiles to analyse this behaviour? UVMMd does require a large amount of memory, especially if EC2 connections are configured. But we did not see a memory-leak in uvmm-daemon itself.
Comment 2 Philipp Hahn univentionstaff 2015-01-22 14:32:35 CET
See Bug #35977 comment 3 for a UVMMd memory leak with form info on debugging that kind of problems using "python-meliae".
Comment 3 Stefan Gohmann univentionstaff 2015-02-23 07:09:51 CET
Reported via feedback: Ticket #2015021821000235
Comment 4 Stefan Gohmann univentionstaff 2015-02-24 06:26:09 CET
More feedback from the partner: They have three KVM servers which are managed through UVMMd. UVMMd runs on two systems.
Comment 5 Philipp Hahn univentionstaff 2015-03-29 11:45:21 CEST
Using meliae I've found that xml.dom.minidom seems to have a memory leak: <http://stackoverflow.com/questions/26787026/memory-leak-parsing-xml-using-xml-dom-minidom>

r59486 | Bug #36640 UVMM: Fix ucslint errors/warning
r59485 | Bug #36640 UVMM: Bump version
r59483 | Bug #36640 UVMM: Fix memory leak in XML parsing
r59482 | Bug #36640 UVMM: Split/add unit test
r59481 | Bug #36640 UVMM: Add memory debugger

Package: univention-virtual-machine-manager-daemon
Version: 4.0.24-2.592.201503291138
Branch: ucs_4.0-0
Scope: errata4.0-1

r59487 | Bug #36635,Bug #36640 UVMM: YAML
Comment 6 Philipp Hahn univentionstaff 2015-03-30 14:32:17 CEST
r59498 | Bug #36640 UVMM: Fix memory leak in XML parsing
 lxml does not allow setting an attribute to None

Package: univention-virtual-machine-manager-daemon
Version: 4.0.24-3.593.201503301426
Branch: ucs_4.0-0
Scope: errata4.0-1

r59499 | Bug #36640 UVMM: Fix memory leak in XML parsing YAML
Comment 7 Erik Damrose univentionstaff 2015-04-02 15:04:01 CEST
OK: r59498 | Bug #36640 UVMM: Fix memory leak in XML parsing
 lxml does not allow setting an attribute to None
-> Creating new instances is once more possible

Reopen: As discussed, the memory leak is still present, as tested with the most recent version on laiva
Comment 8 Philipp Hahn univentionstaff 2015-04-08 12:31:43 CEST
# while cat /proc/$(</etc/runit/univention-virtual-machine-manager-daemon/supervise/pid)/statm;do sleep 1;done
shows VmRSS to increase permanently in fixed intervals.
See <http://xen12.knut.univention.de/collectd/bin/index.cgi?hostname=laiva.knut.univention.de&plugin=processes&timespan=86400&action=show_selection&ok_button=OK> for a long-term trend.

# gdb -p ... -ex 'generate-core-file' ; strings
did show a lot of XML data.

# valgrind --leak-check=full --show-reachable=yes --log-file=val.log val.py
shows XMLDesc() leaking memory.

r14565 | Bug #36640 libvirt: Memory leak
 Upstream commit 4acfb169400497da2a82a849dc8aaa65f88ac6d1
fixed at least one huge memory leak, but the memory usage of UVMMd is still not stable (= increasing).
I will let collectd run some more to get a longer trend as I'm not yet convinced that the problem is fixed now.

$ repo_admin.py --cherrypick -r 4.0 --releasedest 4.0 --dest errata4.0-1 -p libvirt-python

Package: libvirt-python
Version: 1.2.6-2.3.201504081137
Branch: ucs_4.0-0
Scope: errata4.0-1

r59644 | Bug #36640 libvirt-python: Memory leak

r59631 | Bug #36640 UVMM: Fix optional keymap in XML parsing

Package: univention-virtual-machine-manager-daemon
Version: 4.0.24-7.597.201504081138
Branch: ucs_4.0-0
Scope: errata4.0-1
Comment 9 Philipp Hahn univentionstaff 2015-04-09 09:16:29 CEST
Created attachment 6812 [details]
Round-Robin Database: UVMMd Virtual Memory usage

UVMMd still shows a memory leak, but a lot less then before.
# rrdtool graph ps_vm.png -s '09:00 2015-04-07' -e '09:00 2015-04-09' DEF:value=ps_vm.rrd:value:AVERAGE LINE1:value#0000FF:"PsVm"
# display ps_vm.png

If it becomes a problem again, clone this bug for further memory leak debugging:
* Use Valgrind
* Suppress "use-once" allocations <http://valgrind.org/docs/manual/manual-core.html#manual-core.suppress>
Comment 10 Philipp Hahn univentionstaff 2015-04-09 09:17:07 CEST
Created attachment 6813 [details]
Simple test script

The program shows an ever increasing memory usage.
Comment 11 Philipp Hahn univentionstaff 2015-04-10 08:23:37 CEST
(In reply to Philipp Hahn from comment #10)
> Created attachment 6813 [details]
> Simple test script
> The program shows an ever increasing memory usage.

That's wrong: python2.7-dbg has the memory leak - the script is fine, so no easy way to re-produce the leak so far.
Comment 12 Erik Damrose univentionstaff 2015-04-13 16:40:44 CEST
OK: YAML r59644 | Bug #36640 libvirt-python: Memory leak

OK: r59631 | Bug #36640 UVMM: Fix optional keymap in XML parsing

OK: r14565 | Bug #36640 libvirt: Memory leak

-> Verified