Bug 55286 - u-group-membership-cache dir grows in size, files get not cleaned up
u-group-membership-cache dir grows in size, files get not cleaned up
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: General
UCS 5.0
Other Linux
: P5 normal (vote)
: UCS 5.0-2-errata
Assigned To: Philipp Hahn
Dirk Wiesenthal
https://bugs.debian.org/cgi-bin/bugre...
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2022-10-18 18:48 CEST by Erik Damrose
Modified: 2022-11-02 10:38 CET (History)
5 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 7: Crash: Bug causes crash or data loss
Who will be affected by this bug?: 1: Will affect a very few installed domains
How will those affected feel about the bug?: 5: Blocking further progress on the daily work
User Pain: 0.200
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2022090221000239, 2022090121000464
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Erik Damrose univentionstaff 2022-10-18 18:48:53 CEST
We got feedback that the cache files by univention-group-membership-cache are not always cleaned up correctly. Two cases have been observed:

* Unused files with a different extension were found in the cache dir. Manually removing the files fixed the issue.

* File handles of deleted files were not closed by the cache's listener module, filling up the disk. The files were not visible in the directory. Restart of u-directory-listener helped as a workaround.
Comment 1 Erik Damrose univentionstaff 2022-10-18 18:51:25 CEST
2022090121000464
2022090221000239 message #31 for analysis, open file handle issue
Comment 2 Philipp Hahn univentionstaff 2022-10-18 19:15:34 CEST
> Was an der lsof-Ausgabe auffällt, ist dass der Listener jeweils sehr, sehr viele Handles im Status DEL hat, sprich Dateien, die mal geöffnet wurden, dann in der Zwischenzeit gelöscht, aber vom ja noch laufenden Prozess nicht geschlossen:
>
> univentio 2119 listener  DEL-W  REG    254,0             6293029 /usr/share/univention-group-membership-cache/caches/uniqueMembers.db
> univentio 2119 listener  DEL-W  REG    254,0             6293042 /usr/share/univention-group-membership-cache/caches/memberUids.db
Comment 3 Philipp Hahn univentionstaff 2022-10-20 21:37:24 CEST
#!/usr/bin/env python3
import subprocess
import dbm.gnu
DB = "./db.gdbm"
for i in range(3):
  db = dbm.gnu.open(DB, "csu")
  db.reorganize()
  db.close()
  subprocess.call(f"lsof|grep {DB}", shell=True)


<https://bugs.python.org/issue13947> → <https://git.gnu.org.ua/gdbm.git/commit/?id=a0d6c1a8> does NOT fix it.

# dpkg -s libgdbm6
Package: libgdbm6
Source: gdbm
Version: 1.18.1-4

# dpkg -s python3-gdbm
Package: python3-gdbm
Source: python3-stdlib-extensions
Version: 3.7.3-1

<https://packages.debian.org/search?keywords=gdbm&searchon=sourcenames&suite=all&section=all>

<https://univention-dist-binpkg-webgui.k8s.knut.univention.de/source/gdbm/>

Debian-9-Stretch   UCS-4.4  1.8.3-14
Debian-10-Buster   UCS-5.0  1.18.1-4
Debian-11-Bullseye          1.19-2
Debian-12-Bookworm          1.23-3


git bisect start '--term-new' 'fixed' '--term-old' 'broken'
# fixed: [331f05ac9c58d358806fe1bcba88a01467ab0895] Bugfix
git bisect fixed 331f05ac9c58d358806fe1bcba88a01467ab0895
# broken: [778cc81d55aecd6344d577919cec73e4e6980e2e] Version 1.18
git bisect broken 778cc81d55aecd6344d577919cec73e4e6980e2e
# broken: [611cac791f192834d49cd1bd8cfab76a190bfc40] Document new gdbmtool options
git bisect broken 611cac791f192834d49cd1bd8cfab76a190bfc40
# fixed: [3052f2b51eaa9ba047d43ae97581ae1fd895c131] Add missing include
git bisect fixed 3052f2b51eaa9ba047d43ae97581ae1fd895c131
# fixed: [e4089536f849644b12d9647aff12f2dac312b940] Version 1.21 released
git bisect fixed e4089536f849644b12d9647aff12f2dac312b940
# broken: [038872b1582c6b99cc352551875aba0e81f6b5f0] Minor fix
git bisect broken 038872b1582c6b99cc352551875aba0e81f6b5f0
# fixed: [a62815cdf0b60725d6445968ce3d5c39e912eb9c] Update docs
git bisect fixed a62815cdf0b60725d6445968ce3d5c39e912eb9c
# fixed: [cc7051ae2ea384863937083a3a60a5a008d511a5] gdbmshell: get rid of remaining globals
git bisect fixed cc7051ae2ea384863937083a3a60a5a008d511a5
# fixed: [d19407eaa4b00a724c4ff3744c2f49269183da26] gdbmtool: bugfixes
git bisect fixed d19407eaa4b00a724c4ff3744c2f49269183da26
# fixed: [f77b0273db2d1f0cc1ba2d3256acfab1bda1f584] Fix duplicated mmap in gdbm_recover
git bisect fixed f77b0273db2d1f0cc1ba2d3256acfab1bda1f584
# first fixed commit: [f77b0273db2d1f0cc1ba2d3256acfab1bda1f584] Fix duplicated mmap in gdbm_recover

There are too many changes leading up to v1.21~11, so cherry-picking does NOT work.
Comment 4 Philipp Hahn univentionstaff 2022-10-21 22:24:27 CEST
r19719 | Bug #55286 GDBM: Backport FD leak on reorganize
r19720 | Bug #55286 GDBM: Backport FD leak on reorganize
r19721 | Bug #55286 GDBM: Backport FD leak on reorganize

Package: gdbm
Version: 1.18.1-4A~5.0.0.202210212221
Branch: ucs_5.0-0
Scope: errata5.0-2

QA: python3 -c 'import os,subprocess;import dbm.gnu;DB = "./db.gdbm";db = dbm.gnu.open(DB, "csu");db.reorganize();db.close();subprocess.call(f"lsof -p {os.getpid()}|grep {DB}", shell=True)'

TODO: yaml
TODO: more testing
Comment 6 Philipp Hahn univentionstaff 2022-10-25 10:28:02 CEST
[5.0-2] ce0be5ad2c Bug #55286: gdbm 1.18.1-4A~5.0.0.202210212221
 doc/errata/staging/gdbm.yaml | 13 +++++++++++++
 1 file changed, 13 insertions(+)
Comment 7 Dirk Wiesenthal univentionstaff 2022-10-25 11:28:18 CEST
OK: Manual tests including several thousands of changes that run through the listener: Disk space does not grow indefinitely
OK: YAML
OK: Tests
Comment 9 Oliver Friedrich univentionstaff 2022-11-02 10:38:47 CET
Also see https://help.univention.com/t/howto-fix-re-join-in-big-environments/20662
(group permissions of cache files wrong)