Univention Bugzilla – Bug 35977
uvmm libcloud memory consumption
Last modified: 2014-11-26 06:54:22 CET
For the purpose of instance creation we load all available images via libcloud and cache it in the local memory. Just receiving this information uses 400-500MiB of RAM. The uvmmd node should have 2GiB or more RAM available. It may be possible to only store a portion of the acquired information, but during the initial request step this much memory will be needed, as paginated or partial transmission of the list is currently not supported.
Addition: this much ram is needed when requesting ~20000 images
Maybe it is similar to Bug #24273 (r28977).
libcloud internally uses the ElementTree XML API to parse the large response from AWS. If the most efficient C implementation "lxml.etree" from "pyhton-lxml" is not available, it falls back to the Python internal xml.etree.ElementTree implementation, which seems to be a memory hog, as it requires an explicit call to the garbage collector to get the memory freed: scanner.dump_all_objects('1.json') d = ec2driver() images = d.list_images(ex_owner="") scanner.dump_all_objects('2.json') gc.collect() scanner.dump_all_objects('3.json') del images scanner.dump_all_objects('4.json') gc.collect() scanner.dump_all_objects('5.json') del d scanner.dump_all_objects('6.json') gc.collect() scanner.dump_all_objects('7.json') # ./35977_testlibcloud.py ; ls -gGh ?.json -rw-r--r-- 1 4,4M Okt 30 13:58 1.json -rw-r--r-- 1 407M Okt 30 13:59 2.json -rw-r--r-- 1 76M Okt 30 13:59 3.json -rw-r--r-- 1 4,5M Okt 30 13:59 4.json -rw-r--r-- 1 4,5M Okt 30 13:59 5.json -rw-r--r-- 1 4,5M Okt 30 13:59 6.json -rw-r--r-- 1 4,5M Okt 30 13:59 7.json # apt-get install python-lxml # ./35977_testlibcloud.py ; ls -gGh ?.json -rw-r--r-- 1 4,9M Okt 30 14:26 1.json -rw-r--r-- 1 76M Okt 30 14:26 2.json -rw-r--r-- 1 76M Okt 30 14:27 3.json -rw-r--r-- 1 5,0M Okt 30 14:27 4.json -rw-r--r-- 1 5,0M Okt 30 14:27 5.json -rw-r--r-- 1 5,0M Okt 30 14:27 6.json -rw-r--r-- 1 5,0M Okt 30 14:27 7.json # lsof -p 10013 | grep lxml univentio 10013 root mem REG 254,0 1209848 22282824 /usr/lib/python2.7/dist-packages/lxml/etree.so
r55161 | Bug #35977 UVMM: Switch to more efficient python-lxml Package: univention-virtual-machine-manager-daemon Version: 4.0.7-1.538.201410301518 Branch: ucs_4.0-0 No changelog needed as libcloud is new in UCS-4.0
OK: python-lxml added to dependencies OK: no changelog required OK: reduced memory consumption Verified
UCS 4.0-0 has been released: http://docs.univention.de/release-notes-4.0-0-en.html http://docs.univention.de/release-notes-4.0-0-de.html If this error occurs again, please use "Clone This Bug".