Univention Bugzilla – Bug 55286
u-group-membership-cache dir grows in size, files get not cleaned up
Last modified: 2022-11-02 10:38:47 CET
We got feedback that the cache files by univention-group-membership-cache are not always cleaned up correctly. Two cases have been observed: * Unused files with a different extension were found in the cache dir. Manually removing the files fixed the issue. * File handles of deleted files were not closed by the cache's listener module, filling up the disk. The files were not visible in the directory. Restart of u-directory-listener helped as a workaround.
2022090121000464 2022090221000239 message #31 for analysis, open file handle issue
> Was an der lsof-Ausgabe auffällt, ist dass der Listener jeweils sehr, sehr viele Handles im Status DEL hat, sprich Dateien, die mal geöffnet wurden, dann in der Zwischenzeit gelöscht, aber vom ja noch laufenden Prozess nicht geschlossen: > > univentio 2119 listener DEL-W REG 254,0 6293029 /usr/share/univention-group-membership-cache/caches/uniqueMembers.db > univentio 2119 listener DEL-W REG 254,0 6293042 /usr/share/univention-group-membership-cache/caches/memberUids.db
#!/usr/bin/env python3 import subprocess import dbm.gnu DB = "./db.gdbm" for i in range(3): db = dbm.gnu.open(DB, "csu") db.reorganize() db.close() subprocess.call(f"lsof|grep {DB}", shell=True) <https://bugs.python.org/issue13947> → <https://git.gnu.org.ua/gdbm.git/commit/?id=a0d6c1a8> does NOT fix it. # dpkg -s libgdbm6 Package: libgdbm6 Source: gdbm Version: 1.18.1-4 # dpkg -s python3-gdbm Package: python3-gdbm Source: python3-stdlib-extensions Version: 3.7.3-1 <https://packages.debian.org/search?keywords=gdbm&searchon=sourcenames&suite=all§ion=all> <https://univention-dist-binpkg-webgui.k8s.knut.univention.de/source/gdbm/> Debian-9-Stretch UCS-4.4 1.8.3-14 Debian-10-Buster UCS-5.0 1.18.1-4 Debian-11-Bullseye 1.19-2 Debian-12-Bookworm 1.23-3 git bisect start '--term-new' 'fixed' '--term-old' 'broken' # fixed: [331f05ac9c58d358806fe1bcba88a01467ab0895] Bugfix git bisect fixed 331f05ac9c58d358806fe1bcba88a01467ab0895 # broken: [778cc81d55aecd6344d577919cec73e4e6980e2e] Version 1.18 git bisect broken 778cc81d55aecd6344d577919cec73e4e6980e2e # broken: [611cac791f192834d49cd1bd8cfab76a190bfc40] Document new gdbmtool options git bisect broken 611cac791f192834d49cd1bd8cfab76a190bfc40 # fixed: [3052f2b51eaa9ba047d43ae97581ae1fd895c131] Add missing include git bisect fixed 3052f2b51eaa9ba047d43ae97581ae1fd895c131 # fixed: [e4089536f849644b12d9647aff12f2dac312b940] Version 1.21 released git bisect fixed e4089536f849644b12d9647aff12f2dac312b940 # broken: [038872b1582c6b99cc352551875aba0e81f6b5f0] Minor fix git bisect broken 038872b1582c6b99cc352551875aba0e81f6b5f0 # fixed: [a62815cdf0b60725d6445968ce3d5c39e912eb9c] Update docs git bisect fixed a62815cdf0b60725d6445968ce3d5c39e912eb9c # fixed: [cc7051ae2ea384863937083a3a60a5a008d511a5] gdbmshell: get rid of remaining globals git bisect fixed cc7051ae2ea384863937083a3a60a5a008d511a5 # fixed: [d19407eaa4b00a724c4ff3744c2f49269183da26] gdbmtool: bugfixes git bisect fixed d19407eaa4b00a724c4ff3744c2f49269183da26 # fixed: [f77b0273db2d1f0cc1ba2d3256acfab1bda1f584] Fix duplicated mmap in gdbm_recover git bisect fixed f77b0273db2d1f0cc1ba2d3256acfab1bda1f584 # first fixed commit: [f77b0273db2d1f0cc1ba2d3256acfab1bda1f584] Fix duplicated mmap in gdbm_recover There are too many changes leading up to v1.21~11, so cherry-picking does NOT work.
r19719 | Bug #55286 GDBM: Backport FD leak on reorganize r19720 | Bug #55286 GDBM: Backport FD leak on reorganize r19721 | Bug #55286 GDBM: Backport FD leak on reorganize Package: gdbm Version: 1.18.1-4A~5.0.0.202210212221 Branch: ucs_5.0-0 Scope: errata5.0-2 QA: python3 -c 'import os,subprocess;import dbm.gnu;DB = "./db.gdbm";db = dbm.gnu.open(DB, "csu");db.reorganize();db.close();subprocess.call(f"lsof -p {os.getpid()}|grep {DB}", shell=True)' TODO: yaml TODO: more testing
[5.0-2] ce0be5ad2c Bug #55286: gdbm 1.18.1-4A~5.0.0.202210212221 doc/errata/staging/gdbm.yaml | 13 +++++++++++++ 1 file changed, 13 insertions(+)
OK: Manual tests including several thousands of changes that run through the listener: Disk space does not grow indefinitely OK: YAML OK: Tests
<https://errata.software-univention.de/#/?erratum=5.0x468>
Also see https://help.univention.com/t/howto-fix-re-join-in-big-environments/20662 (group permissions of cache files wrong)