Bug 23367 - Build with libdb4.8 or mdb - replace libdb3
Build with libdb4.8 or mdb - replace libdb3
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Listener (univention-directory-listener)
UCS 3.0
Other Linux
: P5 enhancement (vote)
: UCS 4.2
Assigned To: Arvid Requate
Philipp Hahn
https://hutten.knut.univention.de/med...
: interim-2
: 23205 43614 (view as bug list)
Depends on: 36878
Blocks: 43763 43764 43765 44166 44466 50407
  Show dependency treegraph
 
Reported: 2011-08-19 06:34 CEST by Stefan Gohmann
Modified: 2019-10-23 15:54 CEST (History)
5 users (show)

See Also:
What kind of report is it?: Feature Request
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional): API change, Cleanup, Release Goal
Max CVSS v3 score:


Attachments
db_4.8.patch (15.23 KB, patch)
2011-08-19 06:36 CEST, Stefan Gohmann
Details | Diff
switch_listener_to_lmdb.patch (32.98 KB, patch)
2016-08-19 10:22 CEST, Arvid Requate
Details | Diff
switch_listener_to_lmdb.patch (35.50 KB, patch)
2016-08-19 14:18 CEST, Arvid Requate
Details | Diff
swtch_listener_to_lmdb.patch (35.27 KB, patch)
2016-08-19 16:31 CEST, Arvid Requate
Details | Diff
udl-lmdb.patch (88.74 KB, patch)
2017-02-08 20:15 CET, Arvid Requate
Details | Diff
udl-lmdb.patch (90.67 KB, patch)
2017-02-08 23:17 CET, Arvid Requate
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Gohmann univentionstaff 2011-08-19 06:34:38 CEST
Der Build gegen libdb4.8 macht aktuell im Listener einige Probleme: 
 Bug #23160
 Bug #23205.

Ich habe den Patch auf db4.8 zunächst rückgängig gemacht und erneut gegen db3 gebaut.
Comment 1 Stefan Gohmann univentionstaff 2011-08-19 06:36:19 CEST
Created attachment 3455 [details]
db_4.8.patch

Die zurückgenommene db4.8 Anpassung.
Comment 2 Stefan Gohmann univentionstaff 2011-08-19 06:41:06 CEST
*** Bug 23205 has been marked as a duplicate of this bug. ***
Comment 3 Stefan Gohmann univentionstaff 2011-08-19 06:47:08 CEST
*** Bug 23160 has been marked as a duplicate of this bug. ***
Comment 4 Moritz Muehlenhoff univentionstaff 2012-02-14 10:30:18 CET
In Wheezy ist db3 nicht mehr vorhanden, in UCD 3.2 wird die Squeeze-Version vorerst wieder importiert.
Comment 5 Arvid Requate univentionstaff 2016-08-19 10:22:10 CEST
Created attachment 7901 [details]
switch_listener_to_lmdb.patch

The patch adjusts the univention-directory-listener source code to use LMDB instead of BDB. An existing cache.db gets converted (dump+load) in postinst.

The (mandatory) transaction handling probably could be improved for performance, but this patch shows the minimal changes required to make the switch.
Comment 6 Arvid Requate univentionstaff 2016-08-19 14:18:55 CEST
Created attachment 7904 [details]
switch_listener_to_lmdb.patch

The updated patch fixes transaction commit in cache_free_cursor, adds debug messages for the transactions and sets the maxsize to the i386 compatible value that was chosen for Bug #33993. We should probably evaluate the UCR variable ldap/database/mdb/maxsize instead of using a static value here.


Also, the migration via dump&load doesn't work, the resulting cache/data.mdb contains data that univention-directory-listener-dump doesn't accept, I didn't check the details of this issue yet.
Comment 7 Arvid Requate univentionstaff 2016-08-19 16:31:04 CEST
Created attachment 7910 [details]
swtch_listener_to_lmdb.patch

Simplify cache_get_master_entry, no cursor required there.

No clue about the migration yet. Maybe it's a db3 issue? In UCS 4.1 db_dump is from db5.1-util. Running db_upgrade didn't help though.
Comment 8 Arvid Requate univentionstaff 2016-08-22 13:38:01 CEST
I've added the transaction re-use optimization, see git repo URL above.

For the migration I'll simply write a conversion tool. This also helps to address this one issue with MDB:

The keysize in MDB is limited to 511 bytes by default, so it may be clever to use the backend db conversion as an opportunity adjust the key format of the cache. I see two options here:

a) Use the approach of OpenLDAP back-mdb and store DNs hierarchically as a set of ordered RDNs (see http://www.openldap.org/conf/odd-sfo-2003/howard-dev.html ).

b) use the entryUUID as key
Comment 9 Philipp Hahn univentionstaff 2016-08-22 13:50:03 CEST
(In reply to Arvid Requate from comment #8)
> b) use the entryUUID as key

Sadly does not work, as from the notifier (and the tanslog overlay) we only get the DN. The UDL requires the cache to do the lookup DN->UUID itself. Only if that works, UDL prefers the UUID.
Comment 10 Arvid Requate univentionstaff 2017-02-08 20:15:37 CET
Created attachment 8415 [details]
udl-lmdb.patch

This patch is based on 11.0.0-7. In postinst it runs a new tool:

univention-directory-listener-convert
Comment 11 Arvid Requate univentionstaff 2017-02-08 23:17:09 CET
Created attachment 8416 [details]
udl-lmdb.patch

Abort the convert if it has already been done. Obsoleting prior patches.
Comment 12 Arvid Requate univentionstaff 2017-02-09 16:35:08 CET
The original cache.{c,h} have been renamed to cache_bdb.{c,h} and their symbols have been renamed (prefixed by "bdb_"). They are required for the conversion tool univention-directory-listener-convert, which is run once during package update.

The contents of the cache can be dumped as usual via
 univention-directory-listener-dump

Low level inspection is also possible using the lmdb-utils:

apt-get install lmdb-utils
mdb_stat -a /var/lib/univention-directory-listener/cache
mdb_dump -a /var/lib/univention-directory-listener/cache


r76584, r76587, r76589 | Bug #23367
  Use LMDB for listener cache
  Run univention-directory-listener-convert in postinst
  New UCR variable listener/cache/mdb/maxsize

Built as: 11.0.1-3A~4.2.0.201702091616

r76586 | Bug #23367
  Changelog
Comment 13 Arvid Requate univentionstaff 2017-02-10 14:35:21 CET
r76590, r76592, r76604 | Bug #23367
  Fix whitespace and Copyright year
  Restrict access to the mdb cache directory during postinst
  strdup instead of error prone malloc & strcpy


Built as: 11.0.1-6A~4.2.0.201702101423
Comment 14 Philipp Hahn univentionstaff 2017-02-10 14:52:40 CET
(In reply to Arvid Requate from comment #13)
> r76590, r76592, r76604 | Bug #23367
>   Fix whitespace and Copyright year
>   Restrict access to the mdb cache directory during postinst
>   strdup instead of error prone malloc & strcpy
> 
> 
> Built as: 11.0.1-6A~4.2.0.201702101423

Please only set this bug to RESOLVED when you're done with your implementation - otherwise you waste my time doing QA on interim versions.
Comment 15 Arvid Requate univentionstaff 2017-02-10 16:00:04 CET
Sorry, I just checked in feedback from Florian and some minor stuff, go ahead.
Comment 16 Philipp Hahn univentionstaff 2017-02-15 09:05:13 CET
FIXED: ChangeLog → r76678
Comment 17 Philipp Hahn univentionstaff 2017-02-22 15:54:32 CET
*** Bug 43614 has been marked as a duplicate of this bug. ***
Comment 18 Stefan Gohmann univentionstaff 2017-02-23 06:06:27 CET
(In reply to Philipp Hahn from comment #17)
> *** Bug 43614 has been marked as a duplicate of this bug. ***

→ reopen

See also our Jenkins tests:
http://jenkins.knut.univention.de:8080/job/UCS-4.2/job/UCS-4.2-0/job/AutotestUpgrade/SambaVersion=s3,Systemrolle=master/

http://jenkins.knut.univention.de:8080/job/UCS-4.2/job/UCS-4.2-0/job/AutotestUpgrade/SambaVersion=s3,Systemrolle=master/ws/updater.log

univention-directory-listener (11.0.1-14A~4.2.0.201702221256) wird eingerichtet ...
Neue Version der Konfigurationsdatei /etc/init.d/univention-directory-listener wird installiert ...
Neue Version der Konfigurationsdatei /etc/network/if-post-down.d/univention-directory-listener wird installiert ...
Neue Version der Konfigurationsdatei /etc/univention/registry.info/variables/univention-directory-listener.cfg wird installiert ...
Neue Version der Konfigurationsdatei /etc/univention/templates/files/etc/runit/univention-directory-listener/run wird installiert ...
Not updating listener/autostart
File: /etc/logrotate.d/univention-directory-listener
File: /etc/runit/univention-directory-listener/run
insserv: warning: script 'K01univention-directory-notifier' missing LSB tags and overrides
insserv: warning: script 'K02bind9' missing LSB tags and overrides
insserv: warning: script 'K01univention-firewall' missing LSB tags and overrides
insserv: warning: script 'K01univention-system-setup-boot' missing LSB tags and overrides
insserv: warning: script 'K01univention-management-console-web-server' missing LSB tags and overrides
insserv: warning: script 'K01univention-management-console-server' missing LSB tags and overrides
insserv: warning: script 'K01univention-saml' missing LSB tags and overrides
insserv: warning: script 'K01univention-welcome-screen' missing LSB tags and overrides
insserv: warning: script 'K01univention-network-common' missing LSB tags and overrides
insserv: warning: script 'K01univention-maintenance' missing LSB tags and overrides
insserv: warning: script 'K01univention-directory-policy' missing LSB tags and overrides
insserv: warning: script 'S02resize2fs' missing LSB tags and overrides
insserv: warning: script 'univention-management-console-server' missing LSB tags and overrides
insserv: warning: script 'univention-directory-notifier' missing LSB tags and overrides
insserv: warning: script 'univention-system-setup-boot' missing LSB tags and overrides
insserv: warning: script 'univention-saml' missing LSB tags and overrides
insserv: warning: script 'univention-network-common' missing LSB tags and overrides
insserv: warning: script 'univention-firewall' missing LSB tags and overrides
insserv: warning: script 'univention-maintenance' missing LSB tags and overrides
insserv: warning: script 'univention-welcome-screen' missing LSB tags and overrides
insserv: warning: script 'univention-system-setup-boot-prepare' missing LSB tags and overrides
insserv: warning: script 'univention-directory-policy' missing LSB tags and overrides
insserv: warning: script 'bind9' missing LSB tags and overrides
insserv: warning: script 'univention-management-console-web-server' missing LSB tags and overrides
insserv: warning: script 'resize2fs' missing LSB tags and overrides
Not updating listener/debug/level
Not updating listener/freespace
Setting listener/network/protocol
Not updating listener/memberuid/skip
Not updating listener/uniquemember/skip
Calling joinscript 03univention-directory-listener.inst ...
2017-02-22 19:52:08.163355997-05:00 (in joinscript_init)
Joinscript 03univention-directory-listener.inst finished with exitcode 1
Stopping Univention Directory Listener Daemon: univention-directory-listener.
22.02.17 19:52:14.514  DEBUG_INIT
22.02.17 19:52:14.515  LISTENER    ( ERROR   ) : Could not open lock file [/var/lib/univention-directory-listener/cache.lock]
ERROR: univention-directory-listener-convert failed
Starting Univention Directory Listener Daemon: univention-directory-listener.
dpkg: Fehler beim Bearbeiten des Paketes univention-directory-listener (--configure):
 Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück
Comment 19 Arvid Requate univentionstaff 2017-02-23 15:06:45 CET
There you go, cache.lock is owned by root because the UDL code calls cache_lock() before drop_privileges():
================================================================================
Stopping Univention Directory Listener Daemon: univention-directory-listener.
23.11.15 21:39:58.316  DEBUG_INIT
23.11.15 21:39:58.317  LISTENER    ( ERROR   ) : Could not open lock file [/var/lib/univention-directory-listener/cache.lock]
ERROR: univention-directory-listener-convert failed
insgesamt 1024
drwxr-xr-x  6 listener nogroup    4096 Nov 23 21:39 .
drwxr-xr-x 67 root     root       4096 Nov 23 21:34 ..
-rw-------  1 listener nogroup    3392 Nov 23 21:39 .bashrc
drwx------  2 listener nogroup    4096 Nov 23 21:39 cache
-rw-------  1 listener nogroup 1011712 Nov 23 21:26 cache.db
-rw-r--r--  1 listener nogroup       0 Nov 23 16:14 cache.db.lock
-rw-r--r--  1 root     root          0 Nov 23 21:39 cache.lock
drwx------  2 listener nogroup    4096 Nov 23 21:26 handlers
-rw-r--r--  1 listener nogroup       4 Nov 23 21:26 notifier_id
-rw-------  1 listener nogroup     675 Nov 23 21:39 .profile
drwxr-xr-x  2 listener nogroup    4096 Nov 23 21:39 .univention-skel
-rw-------  1 listener nogroup       0 Nov 23 21:39 .univention-skel.lock
drwx------ 14 listener nogroup    4096 Nov 23 21:39 windows-profiles
Starting Univention Directory Listener Daemon: univention-directory-listener.
dpkg: Fehler beim Bearbeiten des Paketes univention-directory-listener (--configure):
 Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück
================================================================================

Fixed packed is building.
Comment 20 Philipp Hahn univentionstaff 2017-03-06 13:29:54 CET
OK: univention-directory-listener-convert
OK: univention-directory-listener-dump
OK: univention-directory-listener-verify -D cn=admin,$(ucr get ldap/base) -w $(</etc/ldap.secret) -b $(ucr get ldap/base)
OK: UCRV listener/cache/mdb/maxsize
OK: changelog
OK: Install from DVD
OK: Upgrade
OK: create-32k-users-in-groups
OK: univention-directory-listener -F -h localhost -p 7389 -x -D $(ucr get ldap/hostdn) -y /etc/machine.secret -m /usr/lib/univention-directory-listener/system -d 2 -i

TODO: Handle cache_update_entry: mdb_put: failed: MDB_MAP_FULL: Environment mapsize limit reached (-30792)
 UDL currently only logs it, but continues running.

TODO: Update <http://docs.software-univention.de/developer-reference-4.1.html#listener:details:cache>

TODO: Bug #43720

FAIL: Valgrind find these multiple times:
==962== Conditional jump or move depends on uninitialised value(s)
==962== Use of uninitialised value of size 8
==962==    at 0x5D77A14: PyObject_Free (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==962==    by 0x5D56900: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==962==    by 0x5D1EB9A: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==962==    by 0x5D2ADD9: ??? (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==962==    by 0x405E59: handler_exec
==962==    by 0x4060BA: handler__update
==962==    by 0x40715F: handler_update
==962==    by 0x4082C1: change_new_modules
==962==    by 0x404B14: main
==962==  Uninitialised value was created by a heap allocation
==962==    at 0x4C28C20: malloc (vg_replace_malloc.c:296)
==962==    by 0x4C2AFCF: realloc (vg_replace_malloc.c:692)
==962==    by 0x40F1D3: write_header
==962==    by 0x40F4C8: unparse_entry
                        cache_update_entry_in_transaction
==962==    by 0x40C3E3: cache_update_or_deleteifunused_entry
                        change_init_module
==962==    by 0x407F37: change_new_modules
==962==    by 0x404B14: main
Comment 21 Arvid Requate univentionstaff 2017-03-06 19:44:14 CET
* applied the memory leak patch you provided, thanks!
* applied the improved logging macro you provided, thanks :-)
* added abort-calls in the code at all places where MDB returns
  something unexpected, like MDB_MAP_FULL
* updated the developer reference


I checked the MDB_MAP_FULL behavior with these values:

A) Abort during conversion:
root@master10:~# ucr set listener/cache/mdb/maxsize=524288 # 512k
root@master10:~# univention-upgrade  ## aborts with:
=============================================================================
23.11.15 21:41:15.829  LISTENER    ( ERROR   ) : cache_update_entry: storing updated entry in database failed
23.11.15 21:41:15.830  LISTENER    ( ERROR   ) : cache.c:513:cache_update_entry mdb_txn_commit: failed: MDB_MAP_FULL: Environment mapsize limit reached (-30792)
Aborted
ERROR: univention-directory-listener-convert failed
=============================================================================

B) Abort during normal operation:
root@master10:~# ucr set listener/cache/mdb/maxsize=1048576 # 1M
root@master10:~# univention-upgrade --ignoreterm --ignoressh --n
root@master10:~# create-32k-users-in-groups
root@master10:~# tail -f /var/log/univention/listener.log
=============================================================================
23.11.15 22:15:06.650  LISTENER    ( PROCESS ) : updating 'cn=nscd0030,cn=uid,cn=temporary,cn=univention,dc=ar41i1,dc=qa' command d
23.11.15 22:15:06.656  LISTENER    ( PROCESS ) : updating 'cn=uidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa' command m
23.11.15 22:15:06.660  LISTENER    ( PROCESS ) : updating 'cn=2056,cn=uidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa' command d
23.11.15 22:15:06.666  LISTENER    ( PROCESS ) : updating 'cn=nscd00,cn=groups,dc=ar41i1,dc=qa' command m
23.11.15 22:15:06.669  LISTENER    ( ERROR   ) : cache_update_entry: storing entry in database failed: cn=nscd00,cn=groups,dc=ar41i1,dc=qa
23.11.15 22:15:06.669  LISTENER    ( ERROR   ) : cache.c:469:cache_update_entry_in_transaction mdb_put: failed: MDB_MAP_FULL: Environment mapsize limit reached (-30792)
23.11.15 22:15:06.669  LISTENER    ( WARN    ) : received signal 6
23.11.15 22:15:13.626  DEBUG_INIT
UNIVENTION_DEBUG_BEGIN  : uldap.__open host=master10.ar41i1.qa port=7389 base=dc=ar41i1,dc=qa
UNIVENTION_DEBUG_END    : uldap.__open host=master10.ar41i1.qa port=7389 base=dc=ar41i1,dc=qa
23.11.15 22:15:16.926  LISTENER    ( PROCESS ) : updating 'cn=nscd00,cn=groups,dc=ar41i1,dc=qa' command m
23.11.15 22:15:16.933  LISTENER    ( ERROR   ) : cache_update_entry: storing entry in database failed: cn=nscd00,cn=groups,dc=ar41i1,dc=qa
23.11.15 22:15:16.933  LISTENER    ( ERROR   ) : cache.c:469:cache_update_entry_in_transaction mdb_put: failed: MDB_MAP_FULL: Environment mapsize limit reached (-30792)
23.11.15 22:15:16.936  LISTENER    ( WARN    ) : received signal 6
=============================================================================
root@master10:~# cat /var/lib/univention-directory-listener/notifier_id 
1791
root@master10:~# grep -A1 1791 /var/lib/univention-ldap/notify/transaction
1791 cn=2056,cn=uidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa d
1792 cn=nscd00,cn=groups,dc=ar41i1,dc=qa m
--
root@master10:~# ucr unset listener/cache/mdb/maxsize

And then replication continues.
Comment 22 Arvid Requate univentionstaff 2017-03-06 21:48:08 CET
I've added an info level log message to make dntree_del_id visible in the logs.

Note to self: Entires(id2dn) = 2 * Entires(id2entry) + 1, due to MDB_DUPSORT and link+node for each (R)DN. Can be seen in the output of mdb_stat. The dnid is always increasing, IDs for deleted objects are currently not reused.
Comment 23 Philipp Hahn univentionstaff 2017-03-07 10:03:56 CET
OK: 77369 77371 77372 77373 77379 77380 77381 77390 77391 77392 77393
OK: chown root /var/lib/univention-directory-listener/cache # abort()
OK: ucsr set listener/cache/mdb/maxsize=500k convert # abort()ls
OK: ucsr unset listener/cache/mdb/maxsize
OK: mdb_stat -a -e /var/lib/univention-directory-listener/cache
OK: <http://jenkins.knut.univention.de:8080/job/UCS-4.2/job/UCS-4.2-0/job/HandbookDeveloper/lastBuild/artifact/webroot/developer-reference-4.2.html#listener:details:cache>
Comment 24 Philipp Hahn univentionstaff 2017-03-07 10:42:28 CET
OK: i386 amd64
Comment 25 Stefan Gohmann univentionstaff 2017-04-04 18:29:33 CEST
UCS 4.2 has been released:
 https://docs.software-univention.de/release-notes-4.2-0-en.html
 https://docs.software-univention.de/release-notes-4.2-0-de.html

If this error occurs again, please use "Clone This Bug".
Comment 26 Arvid Requate univentionstaff 2017-04-26 13:30:32 CEST
Just a quick note, in case somebody wonders about cache file sizes or virtual memory consumption of the LMDB listener implementation:

In UCS 4.1-4 I created 10k users in about 1k groups to check the size of the BDB cache vs LMDB:
==============================================================
root@master10:~# du -sh /var/lib/univention-directory-listener/cache.db                                                                                                        
49M     /var/lib/univention-directory-listener/cache.db
==============================================================

After the update the MDB cache uses about 3x as much space:
==============================================================
root@master10:~# du -sh /var/lib/univention-directory-listener/cache
140M    /var/lib/univention-directory-listener/cache
==============================================================

But that contains unused/freed pages, as can be seen by compacting the database:
==============================================================
root@master10:~# mkdir -p /var/tmp/cache
root@master10:~# /usr/bin/mdb_copy -c \
                 /var/lib/univention-directory-listener/cache \
                 /var/tmp/cache
root@master10:~# du -sh /var/tmp/cache
48M     /var/tmp/cache
==============================================================

See https://symas.com/understanding-lmdb-database-file-sizes-and-memory-utilization/ for an explanation of this general behavior.