Univention Bugzilla – Bug 23367
Build with libdb4.8 or mdb - replace libdb3
Last modified: 2019-10-23 15:54:13 CEST
Der Build gegen libdb4.8 macht aktuell im Listener einige Probleme: Bug #23160 Bug #23205. Ich habe den Patch auf db4.8 zunächst rückgängig gemacht und erneut gegen db3 gebaut.
Created attachment 3455 [details] db_4.8.patch Die zurückgenommene db4.8 Anpassung.
*** Bug 23205 has been marked as a duplicate of this bug. ***
*** Bug 23160 has been marked as a duplicate of this bug. ***
In Wheezy ist db3 nicht mehr vorhanden, in UCD 3.2 wird die Squeeze-Version vorerst wieder importiert.
Created attachment 7901 [details] switch_listener_to_lmdb.patch The patch adjusts the univention-directory-listener source code to use LMDB instead of BDB. An existing cache.db gets converted (dump+load) in postinst. The (mandatory) transaction handling probably could be improved for performance, but this patch shows the minimal changes required to make the switch.
Created attachment 7904 [details] switch_listener_to_lmdb.patch The updated patch fixes transaction commit in cache_free_cursor, adds debug messages for the transactions and sets the maxsize to the i386 compatible value that was chosen for Bug #33993. We should probably evaluate the UCR variable ldap/database/mdb/maxsize instead of using a static value here. Also, the migration via dump&load doesn't work, the resulting cache/data.mdb contains data that univention-directory-listener-dump doesn't accept, I didn't check the details of this issue yet.
Created attachment 7910 [details] swtch_listener_to_lmdb.patch Simplify cache_get_master_entry, no cursor required there. No clue about the migration yet. Maybe it's a db3 issue? In UCS 4.1 db_dump is from db5.1-util. Running db_upgrade didn't help though.
I've added the transaction re-use optimization, see git repo URL above. For the migration I'll simply write a conversion tool. This also helps to address this one issue with MDB: The keysize in MDB is limited to 511 bytes by default, so it may be clever to use the backend db conversion as an opportunity adjust the key format of the cache. I see two options here: a) Use the approach of OpenLDAP back-mdb and store DNs hierarchically as a set of ordered RDNs (see http://www.openldap.org/conf/odd-sfo-2003/howard-dev.html ). b) use the entryUUID as key
(In reply to Arvid Requate from comment #8) > b) use the entryUUID as key Sadly does not work, as from the notifier (and the tanslog overlay) we only get the DN. The UDL requires the cache to do the lookup DN->UUID itself. Only if that works, UDL prefers the UUID.
Created attachment 8415 [details] udl-lmdb.patch This patch is based on 11.0.0-7. In postinst it runs a new tool: univention-directory-listener-convert
Created attachment 8416 [details] udl-lmdb.patch Abort the convert if it has already been done. Obsoleting prior patches.
The original cache.{c,h} have been renamed to cache_bdb.{c,h} and their symbols have been renamed (prefixed by "bdb_"). They are required for the conversion tool univention-directory-listener-convert, which is run once during package update. The contents of the cache can be dumped as usual via univention-directory-listener-dump Low level inspection is also possible using the lmdb-utils: apt-get install lmdb-utils mdb_stat -a /var/lib/univention-directory-listener/cache mdb_dump -a /var/lib/univention-directory-listener/cache r76584, r76587, r76589 | Bug #23367 Use LMDB for listener cache Run univention-directory-listener-convert in postinst New UCR variable listener/cache/mdb/maxsize Built as: 11.0.1-3A~4.2.0.201702091616 r76586 | Bug #23367 Changelog
r76590, r76592, r76604 | Bug #23367 Fix whitespace and Copyright year Restrict access to the mdb cache directory during postinst strdup instead of error prone malloc & strcpy Built as: 11.0.1-6A~4.2.0.201702101423
(In reply to Arvid Requate from comment #13) > r76590, r76592, r76604 | Bug #23367 > Fix whitespace and Copyright year > Restrict access to the mdb cache directory during postinst > strdup instead of error prone malloc & strcpy > > > Built as: 11.0.1-6A~4.2.0.201702101423 Please only set this bug to RESOLVED when you're done with your implementation - otherwise you waste my time doing QA on interim versions.
Sorry, I just checked in feedback from Florian and some minor stuff, go ahead.
FIXED: ChangeLog → r76678
*** Bug 43614 has been marked as a duplicate of this bug. ***
(In reply to Philipp Hahn from comment #17) > *** Bug 43614 has been marked as a duplicate of this bug. *** → reopen See also our Jenkins tests: http://jenkins.knut.univention.de:8080/job/UCS-4.2/job/UCS-4.2-0/job/AutotestUpgrade/SambaVersion=s3,Systemrolle=master/ http://jenkins.knut.univention.de:8080/job/UCS-4.2/job/UCS-4.2-0/job/AutotestUpgrade/SambaVersion=s3,Systemrolle=master/ws/updater.log univention-directory-listener (11.0.1-14A~4.2.0.201702221256) wird eingerichtet ... Neue Version der Konfigurationsdatei /etc/init.d/univention-directory-listener wird installiert ... Neue Version der Konfigurationsdatei /etc/network/if-post-down.d/univention-directory-listener wird installiert ... Neue Version der Konfigurationsdatei /etc/univention/registry.info/variables/univention-directory-listener.cfg wird installiert ... Neue Version der Konfigurationsdatei /etc/univention/templates/files/etc/runit/univention-directory-listener/run wird installiert ... Not updating listener/autostart File: /etc/logrotate.d/univention-directory-listener File: /etc/runit/univention-directory-listener/run insserv: warning: script 'K01univention-directory-notifier' missing LSB tags and overrides insserv: warning: script 'K02bind9' missing LSB tags and overrides insserv: warning: script 'K01univention-firewall' missing LSB tags and overrides insserv: warning: script 'K01univention-system-setup-boot' missing LSB tags and overrides insserv: warning: script 'K01univention-management-console-web-server' missing LSB tags and overrides insserv: warning: script 'K01univention-management-console-server' missing LSB tags and overrides insserv: warning: script 'K01univention-saml' missing LSB tags and overrides insserv: warning: script 'K01univention-welcome-screen' missing LSB tags and overrides insserv: warning: script 'K01univention-network-common' missing LSB tags and overrides insserv: warning: script 'K01univention-maintenance' missing LSB tags and overrides insserv: warning: script 'K01univention-directory-policy' missing LSB tags and overrides insserv: warning: script 'S02resize2fs' missing LSB tags and overrides insserv: warning: script 'univention-management-console-server' missing LSB tags and overrides insserv: warning: script 'univention-directory-notifier' missing LSB tags and overrides insserv: warning: script 'univention-system-setup-boot' missing LSB tags and overrides insserv: warning: script 'univention-saml' missing LSB tags and overrides insserv: warning: script 'univention-network-common' missing LSB tags and overrides insserv: warning: script 'univention-firewall' missing LSB tags and overrides insserv: warning: script 'univention-maintenance' missing LSB tags and overrides insserv: warning: script 'univention-welcome-screen' missing LSB tags and overrides insserv: warning: script 'univention-system-setup-boot-prepare' missing LSB tags and overrides insserv: warning: script 'univention-directory-policy' missing LSB tags and overrides insserv: warning: script 'bind9' missing LSB tags and overrides insserv: warning: script 'univention-management-console-web-server' missing LSB tags and overrides insserv: warning: script 'resize2fs' missing LSB tags and overrides Not updating listener/debug/level Not updating listener/freespace Setting listener/network/protocol Not updating listener/memberuid/skip Not updating listener/uniquemember/skip Calling joinscript 03univention-directory-listener.inst ... 2017-02-22 19:52:08.163355997-05:00 (in joinscript_init) Joinscript 03univention-directory-listener.inst finished with exitcode 1 Stopping Univention Directory Listener Daemon: univention-directory-listener. 22.02.17 19:52:14.514 DEBUG_INIT 22.02.17 19:52:14.515 LISTENER ( ERROR ) : Could not open lock file [/var/lib/univention-directory-listener/cache.lock] ERROR: univention-directory-listener-convert failed Starting Univention Directory Listener Daemon: univention-directory-listener. dpkg: Fehler beim Bearbeiten des Paketes univention-directory-listener (--configure): Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück
There you go, cache.lock is owned by root because the UDL code calls cache_lock() before drop_privileges(): ================================================================================ Stopping Univention Directory Listener Daemon: univention-directory-listener. 23.11.15 21:39:58.316 DEBUG_INIT 23.11.15 21:39:58.317 LISTENER ( ERROR ) : Could not open lock file [/var/lib/univention-directory-listener/cache.lock] ERROR: univention-directory-listener-convert failed insgesamt 1024 drwxr-xr-x 6 listener nogroup 4096 Nov 23 21:39 . drwxr-xr-x 67 root root 4096 Nov 23 21:34 .. -rw------- 1 listener nogroup 3392 Nov 23 21:39 .bashrc drwx------ 2 listener nogroup 4096 Nov 23 21:39 cache -rw------- 1 listener nogroup 1011712 Nov 23 21:26 cache.db -rw-r--r-- 1 listener nogroup 0 Nov 23 16:14 cache.db.lock -rw-r--r-- 1 root root 0 Nov 23 21:39 cache.lock drwx------ 2 listener nogroup 4096 Nov 23 21:26 handlers -rw-r--r-- 1 listener nogroup 4 Nov 23 21:26 notifier_id -rw------- 1 listener nogroup 675 Nov 23 21:39 .profile drwxr-xr-x 2 listener nogroup 4096 Nov 23 21:39 .univention-skel -rw------- 1 listener nogroup 0 Nov 23 21:39 .univention-skel.lock drwx------ 14 listener nogroup 4096 Nov 23 21:39 windows-profiles Starting Univention Directory Listener Daemon: univention-directory-listener. dpkg: Fehler beim Bearbeiten des Paketes univention-directory-listener (--configure): Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück ================================================================================ Fixed packed is building.
OK: univention-directory-listener-convert OK: univention-directory-listener-dump OK: univention-directory-listener-verify -D cn=admin,$(ucr get ldap/base) -w $(</etc/ldap.secret) -b $(ucr get ldap/base) OK: UCRV listener/cache/mdb/maxsize OK: changelog OK: Install from DVD OK: Upgrade OK: create-32k-users-in-groups OK: univention-directory-listener -F -h localhost -p 7389 -x -D $(ucr get ldap/hostdn) -y /etc/machine.secret -m /usr/lib/univention-directory-listener/system -d 2 -i TODO: Handle cache_update_entry: mdb_put: failed: MDB_MAP_FULL: Environment mapsize limit reached (-30792) UDL currently only logs it, but continues running. TODO: Update <http://docs.software-univention.de/developer-reference-4.1.html#listener:details:cache> TODO: Bug #43720 FAIL: Valgrind find these multiple times: ==962== Conditional jump or move depends on uninitialised value(s) ==962== Use of uninitialised value of size 8 ==962== at 0x5D77A14: PyObject_Free (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0) ==962== by 0x5D56900: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0) ==962== by 0x5D1EB9A: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0) ==962== by 0x5D2ADD9: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0) ==962== by 0x405E59: handler_exec ==962== by 0x4060BA: handler__update ==962== by 0x40715F: handler_update ==962== by 0x4082C1: change_new_modules ==962== by 0x404B14: main ==962== Uninitialised value was created by a heap allocation ==962== at 0x4C28C20: malloc (vg_replace_malloc.c:296) ==962== by 0x4C2AFCF: realloc (vg_replace_malloc.c:692) ==962== by 0x40F1D3: write_header ==962== by 0x40F4C8: unparse_entry cache_update_entry_in_transaction ==962== by 0x40C3E3: cache_update_or_deleteifunused_entry change_init_module ==962== by 0x407F37: change_new_modules ==962== by 0x404B14: main
* applied the memory leak patch you provided, thanks! * applied the improved logging macro you provided, thanks :-) * added abort-calls in the code at all places where MDB returns something unexpected, like MDB_MAP_FULL * updated the developer reference I checked the MDB_MAP_FULL behavior with these values: A) Abort during conversion: root@master10:~# ucr set listener/cache/mdb/maxsize=524288 # 512k root@master10:~# univention-upgrade ## aborts with: ============================================================================= 23.11.15 21:41:15.829 LISTENER ( ERROR ) : cache_update_entry: storing updated entry in database failed 23.11.15 21:41:15.830 LISTENER ( ERROR ) : cache.c:513:cache_update_entry mdb_txn_commit: failed: MDB_MAP_FULL: Environment mapsize limit reached (-30792) Aborted ERROR: univention-directory-listener-convert failed ============================================================================= B) Abort during normal operation: root@master10:~# ucr set listener/cache/mdb/maxsize=1048576 # 1M root@master10:~# univention-upgrade --ignoreterm --ignoressh --n root@master10:~# create-32k-users-in-groups root@master10:~# tail -f /var/log/univention/listener.log ============================================================================= 23.11.15 22:15:06.650 LISTENER ( PROCESS ) : updating 'cn=nscd0030,cn=uid,cn=temporary,cn=univention,dc=ar41i1,dc=qa' command d 23.11.15 22:15:06.656 LISTENER ( PROCESS ) : updating 'cn=uidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa' command m 23.11.15 22:15:06.660 LISTENER ( PROCESS ) : updating 'cn=2056,cn=uidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa' command d 23.11.15 22:15:06.666 LISTENER ( PROCESS ) : updating 'cn=nscd00,cn=groups,dc=ar41i1,dc=qa' command m 23.11.15 22:15:06.669 LISTENER ( ERROR ) : cache_update_entry: storing entry in database failed: cn=nscd00,cn=groups,dc=ar41i1,dc=qa 23.11.15 22:15:06.669 LISTENER ( ERROR ) : cache.c:469:cache_update_entry_in_transaction mdb_put: failed: MDB_MAP_FULL: Environment mapsize limit reached (-30792) 23.11.15 22:15:06.669 LISTENER ( WARN ) : received signal 6 23.11.15 22:15:13.626 DEBUG_INIT UNIVENTION_DEBUG_BEGIN : uldap.__open host=master10.ar41i1.qa port=7389 base=dc=ar41i1,dc=qa UNIVENTION_DEBUG_END : uldap.__open host=master10.ar41i1.qa port=7389 base=dc=ar41i1,dc=qa 23.11.15 22:15:16.926 LISTENER ( PROCESS ) : updating 'cn=nscd00,cn=groups,dc=ar41i1,dc=qa' command m 23.11.15 22:15:16.933 LISTENER ( ERROR ) : cache_update_entry: storing entry in database failed: cn=nscd00,cn=groups,dc=ar41i1,dc=qa 23.11.15 22:15:16.933 LISTENER ( ERROR ) : cache.c:469:cache_update_entry_in_transaction mdb_put: failed: MDB_MAP_FULL: Environment mapsize limit reached (-30792) 23.11.15 22:15:16.936 LISTENER ( WARN ) : received signal 6 ============================================================================= root@master10:~# cat /var/lib/univention-directory-listener/notifier_id 1791 root@master10:~# grep -A1 1791 /var/lib/univention-ldap/notify/transaction 1791 cn=2056,cn=uidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa d 1792 cn=nscd00,cn=groups,dc=ar41i1,dc=qa m -- root@master10:~# ucr unset listener/cache/mdb/maxsize And then replication continues.
I've added an info level log message to make dntree_del_id visible in the logs. Note to self: Entires(id2dn) = 2 * Entires(id2entry) + 1, due to MDB_DUPSORT and link+node for each (R)DN. Can be seen in the output of mdb_stat. The dnid is always increasing, IDs for deleted objects are currently not reused.
OK: 77369 77371 77372 77373 77379 77380 77381 77390 77391 77392 77393 OK: chown root /var/lib/univention-directory-listener/cache # abort() OK: ucsr set listener/cache/mdb/maxsize=500k convert # abort()ls OK: ucsr unset listener/cache/mdb/maxsize OK: mdb_stat -a -e /var/lib/univention-directory-listener/cache OK: <http://jenkins.knut.univention.de:8080/job/UCS-4.2/job/UCS-4.2-0/job/HandbookDeveloper/lastBuild/artifact/webroot/developer-reference-4.2.html#listener:details:cache>
OK: i386 amd64
UCS 4.2 has been released: https://docs.software-univention.de/release-notes-4.2-0-en.html https://docs.software-univention.de/release-notes-4.2-0-de.html If this error occurs again, please use "Clone This Bug".
Just a quick note, in case somebody wonders about cache file sizes or virtual memory consumption of the LMDB listener implementation: In UCS 4.1-4 I created 10k users in about 1k groups to check the size of the BDB cache vs LMDB: ============================================================== root@master10:~# du -sh /var/lib/univention-directory-listener/cache.db 49M /var/lib/univention-directory-listener/cache.db ============================================================== After the update the MDB cache uses about 3x as much space: ============================================================== root@master10:~# du -sh /var/lib/univention-directory-listener/cache 140M /var/lib/univention-directory-listener/cache ============================================================== But that contains unused/freed pages, as can be seen by compacting the database: ============================================================== root@master10:~# mkdir -p /var/tmp/cache root@master10:~# /usr/bin/mdb_copy -c \ /var/lib/univention-directory-listener/cache \ /var/tmp/cache root@master10:~# du -sh /var/tmp/cache 48M /var/tmp/cache ============================================================== See https://symas.com/understanding-lmdb-database-file-sizes-and-memory-utilization/ for an explanation of this general behavior.