Bug 35761 - Corrupt pickle cache file prevents uvmmd auto-restart
Corrupt pickle cache file prevents uvmmd auto-restart
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Virtualization - UVMM
UCS 4.2
Other Linux
: P5 normal (vote)
: UCS 4.3-2-errata
Assigned To: Philipp Hahn
Jürn Brodersen
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2014-09-01 12:37 CEST by Philipp Hahn
Modified: 2018-12-05 14:38 CET (History)
1 user (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 1: Will affect a very few installed domains
How will those affected feel about the bug?: 5: Blocking further progress on the daily work
User Pain: 0.143
Enterprise Customer affected?: Yes
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2014090121000247
Bug group (optional):
Max CVSS v3 score:
hahn: Patch_Available+


Attachments
/var/log/univention/virtual-machine-manager-daemon-errors.log.1 (34.85 KB, text/plain)
2014-09-01 12:37 CEST, Philipp Hahn
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hahn univentionstaff 2014-09-01 12:37:13 CEST
Created attachment 6082 [details]
/var/log/univention/virtual-machine-manager-daemon-errors.log.1

Ticket #2014090121000247

In a customer environment the Python process crashed with a memory corruption error after writing a corrupt pickle file:

   7 *** glibc detected *** /usr/bin/python2.6: corrupted double-linked list: 0x0000000001d4e670 ***
 454 *** glibc detected *** /usr/bin/python2.6: free(): invalid pointer: 0x00007f143069ebe0 ***
 881 *** glibc detected *** /usr/bin/python2.6: malloc(): memory corruption: 0x00007f32d0081670 ***

The corrupt pickle file was found using the following command:
cd /var/cache/univention-virtual-machine-manager-daemon
for p in *.pic;do python -c 'import sys;from pickle import load;print sys.argv[1];load(open(sys.argv[1],"r"))' "$p";done

UVMMd should catch the TypeError as well and then also skip loading the cache file:

diff --git a/branches/ucs-3.2/ucs-3.2-3/virtualization/univention-virtual-machine-manager-daemon/src/univention/uvmm/node.py b/branches/ucs-3.2/ucs-3.2-3/virtualization/univention-virtual-machine-manager-daemon/src/univention/uvmm/node.py
index 4a2f715..e758ad8 100644
--- a/branches/ucs-3.2/ucs-3.2-3/virtualization/univention-virtual-machine-manager-daemon/src/univention/uvmm/node.py
+++ b/branches/ucs-3.2/ucs-3.2-3/virtualization/univention-virtual-machine-manager-daemon/src/univention/uvmm/node.py
@@ -545,7 +545,10 @@ class Node(PersistentCached):
                                        try:
                                                cache_file = open(cache_file_name, 'r')
                                                try:
-                                                       xml = cache_file.read()
+                                                       try:
+                                                               xml = cache_file.read()
+                                                       except Exception as ex:
+                                                               raise pickle.PickleError(ex)
                                                finally:
                                                        cache_file.close()
                                                assert xml
Comment 1 Philipp Hahn univentionstaff 2014-11-26 12:36:39 CET
Happened again in a similar form: UVMM was crashing every few minutes. After removing the pickle files /var/cache/univention-virtual-machine-manager/**.pic the crashed were gone.

11 core files and the pickle files available in ~phahn/TICKET/0026*/2014-10-21_coredumps/2014-11-26-*/
Comment 3 Philipp Hahn univentionstaff 2018-11-28 11:32:41 CET
Merged into 4.3-2:

[4.3-2] b01000668f Bug #47956: Merge branch 'phahn/47956-uvmm-corruption' into 4.3-2
[4.3-2] 753a76cca6 Bug #35761 uvmm: Drop corrupt cache files on error
 .../univention-virtual-machine-manager-daemon.yaml      |  3 ++-
 .../runit/univention-virtual-machine-manager-daemon/run |  6 ++++++
 .../debian/changelog                                    |  1 +
 .../src/univention/uvmm/node.py                         | 17 +++++++++++++----
 4 files changed, 22 insertions(+), 5 deletions(-)

Package: univention-virtual-machine-manager-daemon
Version: 7.0.0-14A~4.3.0.201811281123
Version: 7.0.0-14A~4.3.0.201811281129
Branch: ucs_4.3-0
Scope: errata4.3-2

[4.3-2] 2cc8ae5f13 Bug #47956: univention-virtual-machine-manager-daemon 7.0.0-14A~4.3.0.201811281129
 doc/errata/staging/univention-virtual-machine-manager-daemon.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 4 Jürn Brodersen univentionstaff 2018-11-28 18:20:09 CET
systemctl restart univention-virtual-machine-manager-daemon.service -> OK

YAML -> OK
Comment 5 Arvid Requate univentionstaff 2018-12-05 14:38:57 CET
<http://errata.software-univention.de/ucs/4.3/346.html>