Univention Bugzilla – Bug 31725
UCR locking problem
Last modified: 2013-07-15 18:57:06 CEST
Created attachment 5274 [details] testucr.py The python UCR module seems to read sometimes an empty file. I think ucr.load should handle this special case. To reproduce start in one shell for((i=0;i<1000;i++)); do ucr set x=y$i; done and in another shell tesucr.py.
I'm able to reproduce much better it if I use it in this way: for((i=0;i<1000;i++)); do ucr set x$i=y$i; done A solution might be to write it into a temporary file and rename the file. The rename operation is an atomic operation.
(In reply to Stefan Gohmann from comment #1) > I'm able to reproduce much better it if I use it in this way: > for((i=0;i<1000;i++)); do ucr set x$i=y$i; done > > A solution might be to write it into a temporary file and rename the file. > The rename operation is an atomic operation. This has been implemented. I've added a test case in ucs-test: 67ucr_concurrent_set_get - 3.2 ucs-test: r41366 & r41371 - 3.2 fix: r41372 - 3.2 changelog: r41373 - 3.1-1-errata fix: r41374 - 3.1-1-errata YAML: r41375
(In reply to Stefan Gohmann from comment #2) > - 3.2 ucs-test: r41366 & r41371 OK > - 3.2 fix: r41372 FAIL: I observed to different error scenarios with 8.0.7-4.428.201306131342 Create x126 Traceback (most recent call last): File "./testucr", line 21, in <module> testLDAPBase(realBase) File "./testucr", line 7, in testLDAPBase ucr.load() File "/usr/lib/pymodules/python2.6/univention/config_registry/backend.py", line 104, in load reg.load() File "/usr/lib/pymodules/python2.6/univention/config_registry/backend.py", line 304, in load self.__save_file(self.file) File "/usr/lib/pymodules/python2.6/univention/config_registry/backend.py", line 332, in __save_file os.rename(temp_filename, filename) OSError: [Errno 2] No such file or directory Create x127 Create x459 E: your request could not be fulfilled try `univention-config-registry --help` for more information Create x461 > - 3.2 changelog: r41373 OK > - 3.1-1-errata fix: r41374 FAIL: see above > - 3.1-1-errata YAML: r41375 OK, but needs to be updates after the fix
(In reply to Philipp Hahn from comment #3) > > - 3.2 fix: r41372 > FAIL: I observed to different error scenarios with 8.0.7-4.428.201306131342 > > Create x126 > Traceback (most recent call last): > File "./testucr", line 21, in <module> > testLDAPBase(realBase) > File "./testucr", line 7, in testLDAPBase > ucr.load() > File "/usr/lib/pymodules/python2.6/univention/config_registry/backend.py", > line 104, in load > reg.load() > File "/usr/lib/pymodules/python2.6/univention/config_registry/backend.py", > line 304, in load > self.__save_file(self.file) > File "/usr/lib/pymodules/python2.6/univention/config_registry/backend.py", > line 332, in __save_file > os.rename(temp_filename, filename) > OSError: [Errno 2] No such file or directory > Create x127 > > Create x459 > E: your request could not be fulfilled > try `univention-config-registry --help` for more information > Create x461 I changed two points. If the file was already renamed by another process I write the current base*.conf to base*.conf.concurrent_<timestamp>. So they don't get lost. Previous UCR reads from a backup file if UCR was not able to read at least 3 lines. That is a fallback if the file was truncated by an error. This happens all the time, because the base-forced.conf only contains 2 lines if no variable was forced. I changed it to read at least 2 lines. 3.2-0: r41438 3.1-1-errata: r41437 YAML: r41445
(In reply to Stefan Gohmann from comment #4) > I changed two points: > 1. If the file was already renamed by another process I > write the current base*.conf to base*.conf.concurrent_<timestamp>. So they > don't get lost. FAIL: see below > 2. Previous UCR reads from a backup file if UCR was not able to read at least 3 > lines. That is a fallback if the file was truncated by an error. This > happens all the time, because the base-forced.conf only contains 2 lines if > no variable was forced. I changed it to read at least 2 lines. OK > 3.2-0: r41438 OK: 9.0.0-2.429.201306171642 OK: 3.1-1 → 3.2-0 > 3.1-1-errata: r41437 OK: 8.0.7-5.430.201306171642 OK: 3.1-1 → errata3.1-1 → 3.2-0 > YAML: r41445 OK # cat >>/etc/apt/sources.list <<'APT' deb http://omar.knut.univention.de/build2 ucs_3.1-0-errata3.1-2/$(ARCH)/ deb http://omar.knut.univention.de/build2 ucs_3.1-0-errata3.1-2/all/ APT # apt-get update # aptitude install '?installed?source-package(univention-config-registry)' # python testucr & # for((i=0;i<1000;i++)); do ucr set x$i=y$i; done # for((i=0;i<1000;i++)); do ucr unset x$i; done # cat >>/etc/apt/sources.list <<'APT' deb http://omar.knut.univention.de/build2 ucs_3.2-0/$(ARCH)/ deb http://omar.knut.univention.de/build2 ucs_3.2-0/all/ APT # apt-get update # aptitude install '?installed?source-package(univention-config-registry)' # python testucr & # for((i=0;i<1000;i++)); do ucr set x$i=y$i; done # for((i=0;i<1000;i++)); do ucr unset x$i; done inotifywait -m -q /etc/univention -e CLOSE_WRITE,DELETE & OK: ucr set --force foo=bar /etc/univention/ CLOSE_WRITE,CLOSE base-forced.conf.bak.temp /etc/univention/ CLOSE_WRITE,CLOSE base-forced.conf.temp /etc/univention/ CLOSE_WRITE,CLOSE base-forced.conf.lock OK: ucr set --ldap-policy foo=bar W: foo is overridden by scope "forced" /etc/univention/ CLOSE_WRITE,CLOSE base-ldap.conf.bak.temp /etc/univention/ CLOSE_WRITE,CLOSE base-ldap.conf.temp /etc/univention/ CLOSE_WRITE,CLOSE base-ldap.conf.lock FAIL: ucr set --schedule foo=bar W: foo is overridden by scope "forced" /etc/univention/ CLOSE_WRITE,CLOSE base-schedule.conf.bak.temp /etc/univention/ CLOSE_WRITE,CLOSE base-schedule.conf.temp /etc/univention/ CLOSE_WRITE,CLOSE base-schedule.conf.lock /etc/univention/ CLOSE_WRITE,CLOSE base-schedule.conf /etc/univention/ CLOSE_WRITE,CLOSE base-schedule.conf.concurrent_1371550859.48 I can reproduce this by resetting UCR back to its initial state. (The test doesn't trigger 100%, but ~90%) cd /etc/univention while true do rm base-schedule.conf* :>base-schedule.conf ucr set --schedule foo=bar [ -f base-schedule.conf.concurrent_* ] && break done Initially base-forced.conf, base-ldap.conf and base-schedule.conf are empty on my newly cloned VM. _ConfigRegistry.__create_base_conf() is creating these empty files. The <2 lines test still seems wrong?
The observed case is very pathological: It happens only when the alternate conf-files are initially empty AND when a concurrent UCR operation reads the files. The worst case is that the data is stored in the backup file; this is an improvement to the current behavior, where the data would be lost completely. As UCR is scheduled to be re-written anyway, that issues is ignored for this erratum.
(In reply to Philipp Hahn from comment #6) > The observed case is very pathological: It happens only when the alternate > conf-files are initially empty AND when a concurrent UCR operation reads the > files. > The worst case is that the data is stored in the backup file; this is an > improvement to the current behavior, where the data would be lost completely. > > As UCR is scheduled to be re-written anyway, that issues is ignored for this > erratum. Yes, we should fix it with Bug #15720.
http://errata.univention.de/ucs/3.1/130.html
*** Bug 30525 has been marked as a duplicate of this bug. ***
*** Bug 31635 has been marked as a duplicate of this bug. ***