Univention Bugzilla – Bug 55235
[4.4] If MDB is full the Notifier may fill up notifier.log eternally with lots of log messages
Last modified: 2022-10-06 09:46:28 CEST
We need a UCS 4.4 backport +++ This bug was initially created as a clone of Bug #49868 +++ In the case of Ticket#: 2019071621000544 the notifier filled up the notifier.log with over 3000 log entries per second: ============================================================================ 17.07.19 18:15:47.622 DEBUG_INIT 17.07.19 19:10:37.044 TRANSFILE ( ERROR ) : 124790699 ldap_add(): Other (e.g., implementation specific) error 17.07.19 19:10:44.315 DEBUG_INIT 17.07.19 19:10:46.099 TRANSFILE ( WARN ) : 123682201 ldap_add() already exists 17.07.19 19:10:46.100 TRANSFILE ( WARN ) : 123682202 ldap_add() already exists 17.07.19 19:10:46.100 TRANSFILE ( WARN ) : 123682203 ldap_add() already exists 17.07.19 19:10:46.100 TRANSFILE ( WARN ) : 123682204 ldap_add() already exists 17.07.19 19:10:46.101 TRANSFILE ( WARN ) : 123682205 ldap_add() already exists 17.07.19 19:10:46.101 TRANSFILE ( WARN ) : 123682206 ldap_add() already exists 17.07.19 19:10:46.101 TRANSFILE ( WARN ) : 123682207 ldap_add() already exists 17.07.19 19:10:46.102 TRANSFILE ( WARN ) : 123682208 ldap_add() already exists 17.07.19 19:10:46.102 TRANSFILE ( WARN ) : 123682209 ldap_add() already exists [...] 17.07.19 19:16:11.553 TRANSFILE ( WARN ) : 124790698 ldap_add() already exists 17.07.19 19:16:11.553 TRANSFILE ( ERROR ) : 124790699 ldap_add(): Other (e.g., implementation specific) error 17.07.19 19:16:17.092 DEBUG_INIT 17.07.19 19:16:18.907 TRANSFILE ( WARN ) : 123682201 ldap_add() already exists 17.07.19 19:16:18.907 TRANSFILE ( WARN ) : 123682202 ldap_add() already exists 17.07.19 19:16:18.908 TRANSFILE ( WARN ) : 123682203 ldap_add() already exists 17.07.19 19:16:18.908 TRANSFILE ( WARN ) : 123682204 ldap_add() already exists 17.07.19 19:16:18.908 TRANSFILE ( WARN ) : 123682205 ldap_add() already exists [...] ============================================================================ I guess the MDB is full again due to all thos transactions (Bug #49867). The Notifier should be adjusted to not fill up the whole disc storage in a short time.
[4.4-9] c78c77257f doc[UDN]: process transactions per line doc/errata/staging/univention-directory-notifier.yaml | 14 ++++++++++++++ management/univention-directory-notifier/debian/changelog | 8 ++++++++ 2 files changed, 22 insertions(+) [4.4-9] d354564bca style[UDN]: Remove empty lines management/univention-directory-notifier/src/cache.c | 11 -------- management/univention-directory-notifier/src/cache.h | 1 - management/univention-directory-notifier/src/callback.c | 41 +---------------------------- management/univention-directory-notifier/src/network.c | 22 +--------------- management/univention-directory-notifier/src/network.h | 10 +------ management/univention-directory-notifier/src/notify.c | 5 ---- management/univention-directory-notifier/src/notify.h | 7 ----- .../src/univention-directory-notifier.c | 15 +++-------- 8 files changed, 7 insertions(+), 105 deletions(-) [4.4-9] f5d0934a00 fix[UDN]: dynamic client memory leak management/univention-directory-notifier/src/callback.c | 3 +++ 1 file changed, 3 insertions(+) [4.4-9] b192a378e4 fix[UDN]: static client memory leaks management/univention-directory-notifier/src/network.c | 6 ++++++ 1 file changed, 6 insertions(+) [4.4-9] 6999d81847 fix[UDN]: static cache memory leaks management/univention-directory-notifier/src/cache.c | 8 ++++++++ management/univention-directory-notifier/src/cache.h | 2 +- management/univention-directory-notifier/src/univention-directory-notifier.c | 3 +++ 3 files changed, 12 insertions(+), 1 deletion(-) [4.4-9] bb460e6745 style[UDN]: Remove empty debug lines management/univention-directory-notifier/src/callback.c | 10 +++++----- management/univention-directory-notifier/src/network.c | 26 ++++++++++++++------------ management/univention-directory-notifier/src/notify.c | 26 +++++++++++++------------- .../src/univention-directory-notifier.c | 6 +++--- 4 files changed, 35 insertions(+), 33 deletions(-) [4.4-9] b9217495d5 feat[UDN]: Debug to STDERR management/univention-directory-notifier/src/univention-directory-notifier.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) [4.4-9] edda5dd5c3 feat[UDN]: Add signal handlers management/univention-directory-notifier/src/network.c | 22 ++++++++++++++++++++-- .../univention-directory-notifier/src/univention-directory-notifier.c | 6 +++++- 2 files changed, 25 insertions(+), 3 deletions(-) [4.4-9] 6d530f03b0 feat[UDN]: Re-enable security hardening management/univention-directory-notifier/src/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) [4.4-9] d444258dc8 refactor[UDN]: Remove duplicate debug line management/univention-directory-notifier/src/notify.c | 36 +++++++++++++++++------------------- 1 file changed, 17 insertions(+), 19 deletions(-) [4.4-9] 027785b747 refactor[UDN]: Extract validation code management/univention-directory-notifier/src/notify.c | 21 +++------------------ 1 file changed, 3 insertions(+), 18 deletions(-) [4.4-9] dac7cdcc81 refactor[UDN]: Remove unneeded NULL check management/univention-directory-notifier/src/notify.c | 7 ------- 1 file changed, 7 deletions(-) [4.4-9] 033a7badf0 fix[UDN]: Free memory in error case management/univention-directory-notifier/src/notify.c | 1 + 1 file changed, 1 insertion(+) [4.4-9] 61a181fbb0 refactor[UDN]: Remove NULLing management/univention-directory-notifier/src/notify.c | 1 - 1 file changed, 1 deletion(-) [4.4-9] c86522a711 style[UDN]: Remove unneeded return management/univention-directory-notifier/src/notify.c | 2 -- 1 file changed, 2 deletions(-) [4.4-9] d0e44194b9 fix[UDN]: processing 'listener/listener.priv' in smaller chunks management/univention-directory-notifier/src/notify.c | 428 +++++++++++++------------------------- management/univention-directory-notifier/src/notify.h | 6 - 2 files changed, 142 insertions(+), 292 deletions(-) Package: univention-directory-notifier Version: 13.0.3-10A~4.4.0.202210041739 Branch: ucs_4.4-0 Scope: errata4.4-9 [4.4-9] b79662569b Bug #55235: univention-directory-notifier 13.0.3-10A~4.4.0.202210041739 doc/errata/staging/univention-directory-notifier.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) QA: udm users/user create --set firstname=Matthias --set lastname=Hahn --set username=mhahn --set password=univention udm users/user modify --dn uid=mhahn,dc=phahn,dc=dev --set description="$(date)" systemctl stop univention-directory-notifier.service for a in 1 2 3; do udm users/user modify --dn uid=mhahn,dc=phahn,dc=dev --set description="$(date) $$"; done head -n 1 /var/lib/univention-ldap/listener//listener >> /var/lib/univention-ldap/notify/transaction tid=$(tail -n 1 /var/lib/univention-ldap/notify/transaction | cut -d' ' -f1) ./univention-translog load --index $tid tail -f /var/log/univention/notifier.log & systemctl start univention-directory-notifier.service ls -l /var/lib/univention-ldap/listener/ tail -f /var/lib/univention-ldap/notify/transaction /usr/share/univention-directory-notifier/univention-translog stat I had to modify /usr/share/univention-directory-notifier/univention-translog:530 to be able to load $tid < index.count: def load_single(opt): … with Index(opt.translog_file) as index, Translog(opt.translog_file, index) as translog, ldapi(opt) as ld: for tid in opt.trans: try: rec = translog[tid] except IndexError: - continue + if tid < translog.index.count: + continue + translog.seek(translog.index.count) + for rec in translog: + if rec.tid == tid: + break + elif rec.tid < tid: + continue + else: + continue add_transaction(opt, ld, rec)
OK: stopped notifier, did some object modifications. Copied lines from /var/lib/univention-ldap/notify/transaction to /var/lib/univention-ldap/listener/listener, started notifier. Before: Notifier aborts, listener cannot connect to notifier. After: Only new transactions are imported, notifier runs, listener can connect to notifier. OK: Yaml OK: Jenkins Verified
<https://errata.software-univention.de/#/?erratum=4.4x1280>