Univention Bugzilla – Bug 42507
Traceback when reading pickle files
Last modified: 2017-01-05 11:22:40 CET
Bei einem begleiteten Benutzerimport fiel folgender Traceback im 'connector-status.log' auf. --- connect failed, failure was: --- Traceback (most recent call last): File "/usr/share/pyshared/univention/connector/ad/main.py", line 290, in main connect() File "/usr/share/pyshared/univention/connector/ad/main.py", line 234, in connect change_counter=ad.poll_ucs() File "/usr/lib/pymodules/python2.7/univention/connector/__init__.py", line 911, in poll_ucs self._generate_dn_list_from(files) File "/usr/lib/pymodules/python2.7/univention/connector/__init__.py", line 885, in _generate_dn_list_from dn,new,old,old_dn=cPickle.load(f) ValueError: insecure string pickle folgende Pakete sind installiert: python-univention-connector 10.0.2-9.509.201609131411 python-univention-connector-ad 10.0.2-9.509.201609131411 univention-ad-connector 10.0.2-9.509.201609131411
The behavior seems at first to have no adverse effects. A simple try-finally with a file closing statement should easily fix this. 885 dn,new,old,old_dn=cPickle.load(f) 886 if not self.dn_list.get(dn): 887 self.dn_list[dn]=[filename] 888 else: 889 self.dn_list[dn].append(filename) 890
What is a "begleiteten Benutzerimport"? How does it come that the pickle file is corrupt / who created the corrupt pickle file? Is it reproducible? How?
>>> cPickle.loads("Sfoo\np1\n.") Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: insecure string pickle
I think that this is a race condition, not sure. The pickle file is read while it is not completely written to disk?
Hmm, strange, exactly starting with that version (10.0.2-9) the pickle files written by the ad-connector.py listener module are explicitly checked by the module ( via function _dump_changes_to_file_and_check_file ) against corruption. That code has originally been created for the s4-connector.py listener module to address issues of broken pickle files (Bug 29625 and refactored for Bug 24273).
How does it come that this package version was build 2016 but the 2 referenced bugs are from 2012/2013?
Because the fix for the s4-connector has only been merged "later".
(In reply to Arvid Requate from comment #7) > Because the fix for the s4-connector has only been merged "later". So you mean Bug #41938 aka svn r72543
(In reply to Florian Best from comment #2) > What is a "begleiteten Benutzerimport"? > How does it come that the pickle file is corrupt / who created the corrupt > pickle file? > Is it reproducible? How? That means, that I was monitoring an multi Users Import as I noticed this traceback in the connector-status.log, which is normally truncated so the traceback won't never be seen. To reproduce the behavior it's needed to monitor or record the status.log - the debug-level was set to 4, I don't know if this will have any impact to the status.log.
I guess the write of the pickle files is not a atomic process, so they might be incomplete. Do we see any consequences of this error message?
Created attachment 8042 [details] bug_42507.patch The ad-connector.py listener doesn't create the files atomically, so I guess it could happen, that the AD-Connector reads a file that is in the process of creation. The attached patch may be an idea to change that. If this is fixed in the AD-Connector, this bug should be cloned to get fixed in the S4-Connector too.
Created attachment 8054 [details] patch Good idea Arvid! I adapted the patch to use shutil.move (which would support the "tmp" directory to be on a own partition - even if it's only a subdirectory) and use "with open()".
Happened again on UCS version 4.1-3 Errata 291 Wed Oct 12 16:20:37 2016 --- connect failed, failure was: --- Traceback (most recent call last): File "/usr/share/pyshared/univention/connector/ad/main.py", line 290, in main connect() File "/usr/share/pyshared/univention/connector/ad/main.py", line 199, in connect ad.initialize_ucs() File "/usr/lib/pymodules/python2.7/univention/connector/__init__.py", line 822, in initialize_ucs self.poll_ucs() File "/usr/lib/pymodules/python2.7/univention/connector/__init__.py", line 911, in poll_ucs self._generate_dn_list_from(files) File "/usr/lib/pymodules/python2.7/univention/connector/__init__.py", line 885, in _generate_dn_list_from dn,new,old,old_dn=cPickle.load(f) EOFError
Created attachment 8096 [details] check-files.py Attached you find a simple script which checks the pickle files in the directory /var/lib/univention-connector/ad/: root@master161:~# ./check-files.py Please remove the following file(s): /var/lib/univention-connector/ad/.1473765297.412123.swp /var/lib/univention-connector/ad/1473765297.412123 root@master162:~# ./check-files.py Scanned 12 files successfully root@master162:~#
(In reply to Florian Best from comment #12) > Created attachment 8054 [details] > patch Can you add a patch without cleanups?
Created attachment 8311 [details] patch (In reply to Stefan Gohmann from comment #15) > (In reply to Florian Best from comment #12) > > Created attachment 8054 [details] > > patch > > Can you add a patch without cleanups? There hasn't been very much cleanup except the imports which are already fixed now and the 2 times with-open construct. The patch basically is nearly the same length :/
(In reply to Florian Best from comment #16) > Created attachment 8311 [details] > patch Thanks, looks good. Please apply.
Applied that patch. univention-ad-connector (10.0.2-19): r75462 | Bug #42507: Fix race condition between writing and reading pickle files univention-ad-connector.yaml: r75463 | YAML Bug #42507
Missing merge to UCS 4.2: univention-ad-connector (11.0.5-9): r75500 | Bug #42507: Fix race condition between writing and reading pickle files
I added also the commit r75503 from Bug #43235 to keep AD connector and S4 connector code in sync. The removed code block would still work in AD Connector but it also doesn't make sense here. univention-ad-connector (10.0.2-20): r75508 | Bug #42507: code cleanup
Tests: OK Code review: OK UCS Test: OK UCS 4.2 merge: OK YAML: OK
<http://errata.software-univention.de/ucs/4.1/370.html>