Univention Bugzilla – Full Text Bug Listing |
Summary: | uvmm libcloud memory consumption | ||
---|---|---|---|
Product: | UCS | Reporter: | Erik Damrose <damrose> |
Component: | Virtualization - UVMM | Assignee: | Philipp Hahn <hahn> |
Status: | CLOSED FIXED | QA Contact: | Erik Damrose <damrose> |
Severity: | normal | ||
Priority: | P5 | CC: | gohmann, peichert |
Version: | UCS 4.0 | Keywords: | interim-3 |
Target Milestone: | UCS 4.0 | ||
Hardware: | Other | ||
OS: | Linux | ||
See Also: |
https://forge.univention.org/bugzilla/show_bug.cgi?id=35972 https://forge.univention.org/bugzilla/show_bug.cgi?id=36640 |
||
What kind of report is it?: | --- | What type of bug is this?: | --- |
Who will be affected by this bug?: | --- | How will those affected feel about the bug?: | --- |
User Pain: | Enterprise Customer affected?: | ||
School Customer affected?: | ISV affected?: | ||
Waiting Support: | Flags outvoted (downgraded) after PO Review: | ||
Ticket number: | Bug group (optional): | ||
Max CVSS v3 score: |
Description
Erik Damrose
2014-09-22 15:11:27 CEST
Addition: this much ram is needed when requesting ~20000 images Maybe it is similar to Bug #24273 (r28977). libcloud internally uses the ElementTree XML API to parse the large response from AWS. If the most efficient C implementation "lxml.etree" from "pyhton-lxml" is not available, it falls back to the Python internal xml.etree.ElementTree implementation, which seems to be a memory hog, as it requires an explicit call to the garbage collector to get the memory freed: scanner.dump_all_objects('1.json') d = ec2driver() images = d.list_images(ex_owner="") scanner.dump_all_objects('2.json') gc.collect() scanner.dump_all_objects('3.json') del images scanner.dump_all_objects('4.json') gc.collect() scanner.dump_all_objects('5.json') del d scanner.dump_all_objects('6.json') gc.collect() scanner.dump_all_objects('7.json') # ./35977_testlibcloud.py ; ls -gGh ?.json -rw-r--r-- 1 4,4M Okt 30 13:58 1.json -rw-r--r-- 1 407M Okt 30 13:59 2.json -rw-r--r-- 1 76M Okt 30 13:59 3.json -rw-r--r-- 1 4,5M Okt 30 13:59 4.json -rw-r--r-- 1 4,5M Okt 30 13:59 5.json -rw-r--r-- 1 4,5M Okt 30 13:59 6.json -rw-r--r-- 1 4,5M Okt 30 13:59 7.json # apt-get install python-lxml # ./35977_testlibcloud.py ; ls -gGh ?.json -rw-r--r-- 1 4,9M Okt 30 14:26 1.json -rw-r--r-- 1 76M Okt 30 14:26 2.json -rw-r--r-- 1 76M Okt 30 14:27 3.json -rw-r--r-- 1 5,0M Okt 30 14:27 4.json -rw-r--r-- 1 5,0M Okt 30 14:27 5.json -rw-r--r-- 1 5,0M Okt 30 14:27 6.json -rw-r--r-- 1 5,0M Okt 30 14:27 7.json # lsof -p 10013 | grep lxml univentio 10013 root mem REG 254,0 1209848 22282824 /usr/lib/python2.7/dist-packages/lxml/etree.so r55161 | Bug #35977 UVMM: Switch to more efficient python-lxml Package: univention-virtual-machine-manager-daemon Version: 4.0.7-1.538.201410301518 Branch: ucs_4.0-0 No changelog needed as libcloud is new in UCS-4.0 OK: python-lxml added to dependencies OK: no changelog required OK: reduced memory consumption Verified UCS 4.0-0 has been released: http://docs.univention.de/release-notes-4.0-0-en.html http://docs.univention.de/release-notes-4.0-0-de.html If this error occurs again, please use "Clone This Bug". |