Univention Bugzilla – Bug 48427
Notifier and Listener are using an unencrypted and unauthenticated connection
Last modified: 2020-07-23 18:31:36 CEST
I just noticed, that the TCP connection between listener and notifier is completely unencrypted and that the notifier does NOT require any form of authentication. This way, anyone who has access to port 6669 of the affected system/domain has the ability to read out all DNs of the LDAP directory that has been ever used. Since the DNs also contain sensitive data, the following information may be revealed: - primary mail addresses (via cn=temporary) - registered mail domains - names of IMAP shared folders - user names - computer names - IP networks and used IP addresses - SIDs and uidNumbers (via cn=temporary) - installed apps - ... I haven't found extremely critical information like passwords. But a) this is a problem in regard to data protection and privacy. b) this simplifies attacks (e.g. username guessing is not required) c) the data may be used directly for spamming or phishing. Proof of concept: (sleep 1s; echo -e "Version: 2\nCapabilities: \n"; sleep 1s; \ echo -e "MSGID: 1\nGET_ID\n"; sleep 3s) | telnet $HOSTNAME 6669 To get the DN of a known transaction (e.g. 876): (sleep 1s; echo -e "Version: 2\nCapabilities: \n"; sleep 1s; \ echo -e "MSGID: 1\nGET_DN 876\n"; sleep 3s) | telnet $HOSTNAME 6669
As discussed: * Let's add capability level 3 to the notifier, which slightly modifies the current protocol by only sending the change IDs but not any details. * The notifier writes the details into OpenLDAP below a new DN cn=translog (e.g. cn=123,cn=translog), probably via LDAPI. * When the listener sees capability level 3 then it fetches the details from these objects in cn=translog. This happens in an authenticated, TLS-protected LDAP connection. * UCR variables for notifier and listener will allow to set a minimum for the protocol version (to prevent downgrade attacks). In a future UCS version we will raise the default for new installations. * During update we will transfer a reasonably limited number of last transactions from the file to cn=translog (something in the order of 100k). The limit should be adjustable via UCR. When a client asks for an earlier transaction ID while using capability=3, the notifier should signal that it cannot answer the request. * We also need a CLI tool to support pruning old transactions from cn=translog.
CWE-213: Intentional Information Exposure <https://cwe.mitre.org/data/definitions/213.html>
Fixed in branch: <https://git.knut.univention.de/univention/ucs/commits/phahn/48427-udn> Request for review.
I've updated the packages from "deb [trusted=yes] http://10.200.18.180/debian/ phahn48427-udn main" on a UCS 4.3-3 errata407 DC Master. updater.log shows: /var/lib/dpkg/info/univention-directory-notifier.postinst: 117: /var/lib/dpkg/info/univention-directory-notifier.postinst: univention-translog: not found The path is simply missing. Then I did the steps manually: service udn stop; run univention-translog; service udn start. Then I created a user with udm CLI but no transcation is logged, neither to the translog file nor to the new cn=translog.
Created attachment 9808 [details] 1.diff With notifier/debug/level=4 nothing gets logged for a udm users/user create. The notifier hangs in: root@master10:~# strace -p 15326 strace: Process 15326 attached epoll_wait(4, So I guess that's a collateral damage of commit c9141dc51bb. man epoll_wait says: Specifying a timeout of -1 causes epoll_wait() to block indefinitely, while specifying a timeout equal to zero cause epoll_wait() to return immediately, even if no events are available. See attached patch proposals.
Apparently it takes one minute for the notifier to notice the change, I've timed it manually as good as possible: root@master10:~# tail -f /var/lib/univention-ldap/notify/transaction & [...] root@master10:~# date; udm users/user create --set username=user6 --set lastname=last6 --set password=univention Mi 21. Nov 18:44:55 CET 2018 Object created: uid=user6,dc=ar41i1,dc=qa root@master10:~# root@master10:~# 1370 cn=user6,cn=uid,cn=temporary,cn=univention,dc=ar41i1,dc=qa a 1371 cn=2016,cn=uidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa a 1372 cn=2016,cn=gidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa a 1373 cn=2016,cn=gidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa d 1374 uid=user6,dc=ar41i1,dc=qa a 1375 cn=uidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa m 1376 cn=2016,cn=uidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa d 1377 cn=user6,cn=uid,cn=temporary,cn=univention,dc=ar41i1,dc=qa d 1378 uid=user6,dc=ar41i1,dc=qa m 1379 uid=user6,dc=ar41i1,dc=qa m 1380 cn=Domain Users,cn=groups,dc=ar41i1,dc=qa m root@master10:~# date Mi 21. Nov 18:45:55 CET 2018
(In reply to Arvid Requate from comment #4) > I've updated the packages from "deb [trusted=yes] > http://10.200.18.180/debian/ phahn48427-udn main" on a UCS 4.3-3 errata407 > DC Master. updater.log shows: > > /var/lib/dpkg/info/univention-directory-notifier.postinst: 117: > /var/lib/dpkg/info/univention-directory-notifier.postinst: > univention-translog: not found [phahn/48427-udn] 61a275c405 fixup! Bug #48427 udn: Load transaction log into LDAP .../debian/univention-directory-notifier.postinst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (In reply to Arvid Requate from comment #5) > root@master10:~# strace -p 15326 > strace: Process 15326 attached > epoll_wait(4, > > > So I guess that's a collateral damage of commit c9141dc51bb. man epoll_wait > says: > > Specifying a timeout of -1 > causes epoll_wait() to block indefinitely, while specifying a timeout > equal to zero cause epoll_wait() to return immediately, even if no > events are available. > > See attached patch proposals. NO! This would create a busy-wait-loop. The correct fix is [phahn/48427-udn] 893fb9db5e fixup! Bug #48467 udn: Replace select() with epoll() management/univention-directory-notifier/src/network.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (In reply to Arvid Requate from comment #6) > Apparently it takes one minute for the notifier to notice the change, I've > timed it manually as good as possible: I will have a look.
The current plan is to release the new notifier/listener plus LDAP schema extension in KW 7 (ca. 11.-13.2.). Extensive testing is required for an errata update, e.g. because we currently don't enforce that the DC Master is updated before the replicating servers. Special scenarios like that require extra care. In addition to implementing a solution for this bug Philipp also did a security audit for the notifier code and collected a couple of additional improvement proposals which we will address in a prioritized order.
* update on master works but not sure about this output (i think at least the set -x should be removed) univention-ldap-server (14.0.2-35A~~2019-01-29-47721.6e10111f1d71a728a85a9ec241f6d4d27a06af06) wird eingerichtet ... ... Not updating ldap/shadowbind/ignorefilter Not updating ldap/maxopenfiles Not updating ldap/acl/read/anonymous [ ok ] Stopping slapd (via systemctl): slapd.service. 2019-01-29 13:33:20.181990661+01:00 (in joinscript_init) Multifile: /etc/ldap/slapd.conf CRITICAL:__main__:Error in OpenLDAP configuration: 5c504812 WARNING: No dynamic config support for overlay translog. 5c504812 WARNING: No dynamic config support for overlay shadowbind. 5c504812 mdb_db_open: database "cn=translog" cannot be opened: No such file or directory (2). Restore from backup! 5c504812 backend_startup_one (type=mdb, suffix="cn=translog"): bi_db_open failed! (2) slap_startup failed (test would succeed using the -u switch) + trap cleanup EXIT + main db + local action=db + shift + setup_db + local db + dpkg --print-architecture + db=data.mdb + [ -f /var/lib/univention-ldap/translog/data.mdb ] + slapadd -f /etc/ldap/slapd.conf -b cn=translog -l /usr/share/univention-ldap/translog.ldif _#################### 100.00% eta none elapsed none fast! Closing DB... + : + cleanup + local rc=0 + [ 0 -ne 0 ] + return 0 2019-01-29 13:33:22.457124412+01:00 (in joinscript_save_current_version) 2019-01-29 13:33:22.470492866+01:00 (in joinscript_init) [ ok ] Stopping slapd (via systemctl): slapd.service. [ ok ] Starting slapd (via systemctl): slapd.service. [ ok ] Restarting univention-directory-listener (via systemctl): univention-directory-listener.service. * notifier/protocol/version=2 for updates and installation (otherwise th installation of new backup system with notifier/protocol/version=3 could lead to confusion * univention-join, translog copy goes like this: slapd start && ssh master slapcat | slapadd Better to stop the slapd before slapdd? * in my setup i have 4000 users, the translog db on the master has 72000 (file: 66M) entries, during the update of my backup univention-translog created a translog database with 20000 entries (22M) with no notable delay, please consider increasing the minimum (count, size, ...) for the translog history * /usr/share/univention-directory-notifier/univention-translog -h Traceback (most recent call last): File "/usr/share/univention-directory-notifier/univention-translog", line 564, in <module> exit(main()) File "/usr/share/univention-directory-notifier/univention-translog", line 302, in main opt = parse_args() File "/usr/share/univention-directory-notifier/univention-translog", line 495, in parse_args opt = parser.parse_args(args) File "/usr/lib/python2.7/argparse.py", line 1701, in parse_args args, argv = self.parse_known_args(args, namespace) File "/usr/lib/python2.7/argparse.py", line 1733, in parse_known_args namespace, args = self._parse_known_args(args, namespace) File "/usr/lib/python2.7/argparse.py", line 1939, in _parse_known_args start_index = consume_optional(start_index) File "/usr/lib/python2.7/argparse.py", line 1879, in consume_optional take_action(action, args, option_string) File "/usr/lib/python2.7/argparse.py", line 1807, in take_action action(self, namespace, argument_values, option_string) File "/usr/lib/python2.7/argparse.py", line 996, in __call__ parser.print_help() File "/usr/lib/python2.7/argparse.py", line 2340, in print_help self._print_message(self.format_help(), file) File "/usr/lib/python2.7/argparse.py", line 2314, in format_help return formatter.format_help() File "/usr/lib/python2.7/argparse.py", line 281, in format_help help = self._root_section.format_help() File "/usr/lib/python2.7/argparse.py", line 211, in format_help func(*args) File "/usr/lib/python2.7/argparse.py", line 211, in format_help func(*args) File "/usr/lib/python2.7/argparse.py", line 517, in _format_action help_text = self._expand_help(action) File "/usr/lib/python2.7/argparse.py", line 603, in _expand_help return self._get_help_string(action) % params KeyError: 'defauls'
(In reply to Felix Botner from comment #9) > * update on master works but not sure about this output > (i think at least the set -x should be removed) [phahn/48427-udn] 1ff9b8f695 squash! Bug #48427 ldap: Add transaction log database > * notifier/protocol/version=2 for updates and installation (otherwise th > installation of new backup system with notifier/protocol/version=3 could > lead to confusion [phahn/48427-udn] a8739ba933 squash! Bug #48427 udn: Add protocol version 3 > * univention-join, translog copy goes like this: > slapd start && ssh master slapcat | slapadd > Better to stop the slapd before slapdd? [phahn/48427-udn] 40ab694544 squash! Bug #48427 join: Copy translog.mdb on Backup join > * in my setup i have 4000 users, the translog db on the master has 72000 > (file: 66M) entries, during the update of my backup univention-translog > created a translog database with 20000 entries (22M) with no notable delay, > please consider increasing the minimum (count, size, ...) for the translog > history [phahn/48427-udn] d301d0319d squash! Bug #48427 udn: Load transaction log into LDAP > * /usr/share/univention-directory-notifier/univention-translog -h > Traceback (most recent call last): > KeyError: 'defauls' [phahn/48427-udn] 84b369b5a7 squash! Bug #48427 udn: Load transaction log into LDAP
* backup re-join _#################### 100.00% eta none elapsed none fast! Closing DB... unknown option -- - usage: ssh [-1246AaCfGgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec] [-D [bind_address:]port] [-E log_file] [-e escape_char] [-F configfile] [-I pkcs11] [-i identity_file] [-J [user@]host[:port]] [-L address] [-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port] [-Q query_option] [-R address] [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]] [user@]hostname [command] Configure 04univention-ldap-client.inst furthermore @backup-> univention-ssh /tmp/univention \ Administrator@master.four.three /usr/sbin/slapcat 5c51d3a8 /etc/ldap/slapd.conf: line 139: invalid path: Permission denied slapcat: bad configuration file! seems that the admin user is not allowed to slapcat the master * /tmp/transaction from internal 32bit server -> usr/share/univention-directory-notifier/univention-translog --translog /tmp/transaction stat Traceback (most recent call last): File "/usr/share/univention-directory-notifier/univention-translog", line 564, in <module> exit(main()) File "/usr/share/univention-directory-notifier/univention-translog", line 315, in main return opt.func(opt, ld) or 0 File "/usr/share/univention-directory-notifier/univention-translog", line 397, in show_stat with Index(opt.translog) as index, Translog(opt.translog, index) as translog: File "/usr/share/univention-directory-notifier/univention-translog", line 90, in __enter__ assert reminder == 0, reminder AssertionError: 8 maybe this is a 32/64 bit problem (univention-translog was started on 64bit), but Paketlisten werden gelesen... Fertig N: Skipping acquire of configured file 'main/binary-i386/Packages' as repository 'http://10.200.18.180/debian phahn48427-udn InRelease' doesn't support architecture 'i386' So, TODO: i386 tests after merge to master!
(In reply to Felix Botner from comment #11) > > * /tmp/transaction from internal 32bit server > > -> usr/share/univention-directory-notifier/univention-translog --translog > /tmp/transaction stat > Traceback (most recent call last): > File "/usr/share/univention-directory-notifier/univention-translog", line > 564, in <module> > exit(main()) > File "/usr/share/univention-directory-notifier/univention-translog", line > 315, in main > return opt.func(opt, ld) or 0 > File "/usr/share/univention-directory-notifier/univention-translog", line > 397, in show_stat > with Index(opt.translog) as index, Translog(opt.translog, index) as > translog: > File "/usr/share/univention-directory-notifier/univention-translog", line > 90, in __enter__ > assert reminder == 0, reminder > AssertionError: 8 > > maybe this is a 32/64 bit problem (univention-translog was started on > 64bit), but > > Paketlisten werden gelesen... Fertig > N: Skipping acquire of configured file 'main/binary-i386/Packages' as > repository 'http://10.200.18.180/debian phahn48427-udn InRelease' doesn't > support architecture 'i386' > > So, TODO: i386 tests after merge to master! I think we should make univention-translog architecture independent (especially if we fix the join to copy the transaction file instead of slapcat the translog database) Do we really have to "parse" the index file? Could we not just get the last n lines from the transaction file?
(In reply to Felix Botner from comment #11) > * backup re-join As discussed with Arvid "univention-join" will use the downloaded transaction.log text file and import it using "univention-translog import" from UDN. That way no extra file needs to be downloaded. Another option would have been to change the ACL to allow "Domain Admins" and/or "DC Backup Hosts" read access to "cn=translog". > unknown option -- - I have no idea where this comes from, but it should be gone. > /etc/ldap/slapd.conf: line 139: invalid path: Permission denied Same as above. > * /tmp/transaction from internal 32bit server ... > maybe this is a 32/64 bit problem (univention-translog was started on > 64bit), but The index files are architecture dependent. (In reply to Felix Botner from comment #12) > I think we should make univention-translog architecture independent > (especially if we fix the join to copy the transaction file instead of > slapcat the translog database) The script is architecture independent, but the files it works on are not. Similar you cannot copy a LDAP database and expect the slap* tools to work on them. > Do we really have to "parse" the index file? > Could we not just get the last n lines from the transaction file? Yes, one can, but simply stop copying architecture dependent files between incompatible architectures. One of the reasons I started writing "univention-translog" is to add more sub-commands like "rebuild-index" and "check" to make it easier to work with those architecture dependent things. Another reason was that I already had "fun" with that format during development, but as I was asked to not work on Bug #41687 that code will remain hidden in my branch.
[4.3-3] d83f374029 Bug #48427: Merge branch 'phahn/48427-udn' into 4.3-3 Package: ucs-test Version: 8.0.28-261A~4.3.0.201902051137 Branch: ucs_4.3-0 Scope: errata4.3-3 Package: univention-directory-notifier Version: 12.0.1-4A~4.3.0.201902051143 Branch: ucs_4.3-0 Scope: errata4.3-3 Package: univention-directory-listener Version: 12.0.0-34A~4.3.0.201902051144 Branch: ucs_4.3-0 Scope: errata4.3-3 Package: univention-join Version: 10.0.0-26A~4.3.0.201902051146 Branch: ucs_4.3-0 Scope: errata4.3-3 Package: univention-ldap Version: 14.0.2-36A~4.3.0.201902051148 Branch: ucs_4.3-0 Scope: errata4.3-3 Package: univention-management-console-module-diagnostic Version: 4.0.1-21A~4.3.0.201902051151 Branch: ucs_4.3-0 Scope: errata4.3-3 [4.3-3] 2e06fe54c5 Bug #48427 udn,udl,ldap,umc-diag: YAML doc/errata/staging/univention-directory-listener.yaml | 2 +- doc/errata/staging/univention-directory-notifier.yaml | 2 +- doc/errata/staging/univention-join.yaml | 2 +- doc/errata/staging/univention-ldap.yaml | 2 +- doc/errata/staging/univention-management-console-module-diagnostic.yaml | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-)
FAIL - rejoin (backup) with broken translog database FAIL - redirect slapadd output in univention-join OK - Update Tests (M)asetr, (B)ackup, (S)lave, M(E)mber -> update order M B S E B S E M M S E B S E B M After every system update, the replication for all systems has been tested (with all notifiers in the domain) OK - New installation OK - Rejoin
tested with transaction file from out internal master -> /usr/share/univention-directory-notifier/univention-translog -v -t /opt/transaction -l import ... [('objectClass', ['auditObject']), ('reqDN', ['cn=knut,cn=thinclient,cn=policies,dc=knut,dc=univention,dc=de']), ('reqStart', ['20190205141726Z']), ('reqSession', ['20595']), ('reqType', ['\xd9'])] 2019-02-05 14:17:37,761:CRITICAL:ldap_add(reqSession=20595,cn=translog): {'info': 'reqType: value #0 invalid per syntax', 'desc': 'Invalid syntax'} and aborted after this error.
(In reply to Felix Botner from comment #16) > tested with transaction file from out internal master > > -> /usr/share/univention-directory-notifier/univention-translog -v -t > /opt/transaction -l import > ... > > [('objectClass', ['auditObject']), ('reqDN', > ['cn=knut,cn=thinclient,cn=policies,dc=knut,dc=univention,dc=de']), > ('reqStart', ['20190205141726Z']), ('reqSession', ['20595']), ('reqType', > ['\xd9'])] > > 2019-02-05 14:17:37,761:CRITICAL:ldap_add(reqSession=20595,cn=translog): > {'info': 'reqType: value #0 invalid per syntax', 'desc': 'Invalid syntax'} > > and aborted after this error. transaction file corrupt 20595 cn=knut,cn=thinclient,cn=policies,dc=knut,dc=univention,dc=de Ù we should ignore this error (add a "m" instead, or ignore this transaction)
We should reconsider the minima for the "import" command in univention-translog. Currently the tool wants to "convert" at least 100MB of the transaction file, this seems too much: -> wc -l transaction 265472 transaction -> du -h transaction 25M transaction -> time /usr/share/univention-directory-notifier/univention-translog -t /opt/transaction -l import real 3m49,832s -> du -h /var/lib/univention-ldap/translog/ 261M /var/lib/univention-ldap/translog/ so with the 100MB minimum we would convert a million transaction, which takes 16min during the update and join and would result in a 1G ldap database Better take the last 100000 transactions and that's it.
(In reply to Felix Botner from comment #16) > tested with transaction file from out internal master which contains invalid UTF-8 byte sequences; univention-translog will print an error message, skip that transaction and continue with the next transaction (when called with option --lenient). (The "command" we could fix to "m", but not the DN.) (In reply to Felix Botner from comment #18) > We should reconsider the minima for the "import" command in > univention-translog. Changed to 10M Also print a progress report to STDERR when called with a TTY. [4.3-3] f96530b313 Bug #48427: Merge branch 'phahn/48427-udn' into 4.3-3 [4.3-3] 38b7ec4e01 Bug #48427 udn: Skip invalid transactions .../univention-directory-notifier/debian/changelog | 6 +++++ .../univention-translog | 28 +++++++++++++++++----- 2 files changed, 28 insertions(+), 6 deletions(-) [4.3-3] d0f8cc581b Bug #48427: Improve setup-translog management/univention-join/debian/changelog | 6 ++ management/univention-join/univention-join | 12 +--- .../01univention-ldap-server-init.inst | 2 +- management/univention-ldap/debian/changelog | 6 ++ .../debian/univention-ldap-server.postinst | 2 +- management/univention-ldap/scripts/setup-translog | 68 +++++++++++++++------- 6 files changed, 65 insertions(+), 31 deletions(-) Package: univention-join Version: 10.0.0-27A~4.3.0.201902051657 Branch: ucs_4.3-0 Scope: errata4.3-3 Package: univention-ldap Version: 14.0.2-37A~4.3.0.201902051659 Branch: ucs_4.3-0 Scope: errata4.3-3 Package: univention-directory-notifier Version: 12.0.1-6A~4.3.0.201902051701 Branch: ucs_4.3-0 Scope: errata4.3-3 [4.3-3] 061655d491 Bug #48427 udn,ldap,join: YAML doc/errata/staging/univention-directory-notifier.yaml | 2 +- doc/errata/staging/univention-join.yaml | 2 +- doc/errata/staging/univention-ldap.yaml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
why is that? Package: univention-join +Breaks: univention-ldap-server (<< 14.0.2-35)
(In reply to Felix Botner from comment #20) > why is that? > > > Package: univention-join > +Breaks: univention-ldap-server (<< 14.0.2-35) sorry, yes the cn=translog configuration but the rejoin with an broken translog database still fails univention-join 10.0.0-27A~4.3.0.20190205165 univention-directory-listener 12.0.0-34A~4.3.0.201 univention-directory-notifier 12.0.1-7A~4.3.0.201902051728 @backup-> univention-join ... Configure 02univention-directory-notifier.inst done Configure 03univention-directory-listener.ins @backup -> tail -100 /var/log/univention/join.log 05.02.19 18:38:41.021 LISTENER ( WARN ) : replication: Can't contact LDAP server: Can not connect LDAP Server, retry in 10 seconds 05.02.19 18:38:51.033 LISTENER ( WARN ) : replication: Can't contact LDAP server: Can not connect LDAP Server, retry in 10 seconds 05.02.19 18:39:01.039 LISTENER ( WARN ) : replication: Can't contact LDAP server: Can not connect LDAP Server, retry in 10 se -> mdb_stat /var/lib/univention-ldap/translog/ mdb_env_open failed, error -30793 MDB_INVALID: File is not an LMDB file
I got an error message during the update: ''' Multifile: /etc/ldap/slapd.conf CRITICAL:__main__:Error in OpenLDAP configuration: 5c5ab162 WARNING: No dynamic config support for overlay translog. 5c5ab162 WARNING: No dynamic config support for overlay shadowbind. 5c5ab162 mdb_db_open: database "cn=translog" cannot be opened: No such file or directory (2). Restore from backup! 5c5ab162 backend_startup_one (type=mdb, suffix="cn=translog"): bi_db_open failed! (2) slap_startup failed (test would succeed using the -u switch) ''' But after the update slapd was running and cn=translog looked fine. So I guess this can be ignored?
(In reply to Jürn Brodersen from comment #22) > I got an error message during the update: > ''' > Multifile: /etc/ldap/slapd.conf > CRITICAL:__main__:Error in OpenLDAP configuration: > 5c5ab162 WARNING: No dynamic config support for overlay translog. > 5c5ab162 WARNING: No dynamic config support for overlay shadowbind. > 5c5ab162 mdb_db_open: database "cn=translog" cannot be opened: No such file > or directory (2). Restore from backup! > 5c5ab162 backend_startup_one (type=mdb, suffix="cn=translog"): bi_db_open > failed! (2) > slap_startup failed (test would succeed using the -u switch) > ''' > > But after the update slapd was running and cn=translog looked fine. So I > guess this can be ignored? Sorry already reported in comment 9
please add this +++ b/management/univention-directory-notifier/univention-translog @@ -390,7 +390,7 @@ def ldapi(opt): log.debug("ldap_set_option(PROTOCOL_VERSION)") ldap.set_option(ldap.OPT_PROTOCOL_VERSION, ldap.VERSION3) log.debug("ldap_initialize(%s)", opt.ldap) - ld = ReconnectLDAPObject(opt.ldap, trace_level=max(0, opt.verbose - 4)) + ld = ReconnectLDAPObject(opt.ldap, trace_level=max(0, opt.verbose - 4), retry_max=3, retry_delay=10.0) log.debug("ldap_bind()") ld.sasl_external_bind_s() currently the python-ldap reconnect object is not configured and a simple restart of the slapd kills the script (and due to Bug #48530, this is what happens during the update)
(In reply to Felix Botner from comment #24) > please add this ...> currently the python-ldap reconnect object is not configured and a simple > restart of the slapd kills the script (and due to Bug #48530, this is what > happens during the update) [4.3-3] 57604d5694 Bug #48427 udn: Fine-tune ReconnectLDAPObject() management/univention-directory-notifier/univention-translog | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) [4.3-3] a7a92ad1db Bug #48427 udn: Fix Python format strings management/univention-directory-notifier/debian/changelog | 6 ++++++ management/univention-directory-notifier/univention-translog | 6 +++--- 2 files changed, 9 insertions(+), 3 deletions(-) (In reply to Felix Botner from comment #21) > but the rejoin with an broken translog database still fails This should be fixed by this commit, but my VMs currently fail to test it: [4.3-3] 0dbc0f283b Bug #48427 join: Move code to force-reset translog DB. management/univention-join/debian/changelog | 6 ++++++ management/univention-join/univention-join | 8 +++++--- 2 files changed, 11 insertions(+), 3 deletions(-) Package: univention-join Version: 10.0.0-28A~4.3.0.201902061301 Branch: ucs_4.3-0 Scope: errata4.3-3 Package: univention-directory-notifier Version: 12.0.1-8A~4.3.0.201902061304 Branch: ucs_4.3-0 Scope: errata4.3-3 [4.3-3] f63c6c4c70 Bug #48427 udl,join: YAML doc/errata/staging/univention-directory-notifier.yaml | 2 +- doc/errata/staging/univention-join.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
please remove (log to join.log) the univention-translog output in univention-join -> univention-join ... Configure 01univention-ldap-server-init.inst done Configure 02univention-directory-notifier.inst done Configure 03univention-directory-listener.inst done processed 54124 of 60597 [89.3%] in 60.0s, 6473 remaining in 7.2s
OK - join with broken translog database OK - logging in univention-translog OK - ldap reconnect in univention-translog
(In reply to Felix Botner from comment #26) > please remove (log to join.log) the univention-translog output in > univention-join > > -> univention-join > ... > processed 54124 of 60597 [89.3%] in 60.0s, 6473 remaining in 7.2s This is the progress report from univention-translog. You only get it if STDERR is a valid terminal which indicates, that the user invoked it by hand (and not via some script or with some pipe redirection). I consider it a major pain to get NO progress information while somethings happens in the background, especially if you don't know how long it will take. We already scratched our heads that it takes longer even on modern PCs with SSDs than with our VMs using cache="unsafe", so think about users with slow PCs or many transactions to import. They will appreciate that progress information. If you prefer "flights in the dark", append "2>/dev/null" or "2>&1 | tee log" or "2>&1 | cat".
(In reply to Philipp Hahn from comment #28) > > If you prefer "flights in the dark", append "2>/dev/null" or "2>&1 | tee > log" or "2>&1 | cat". this is the way univention-join operates, we have no progress in any of the join scripts, if you want detailed information, open the join.log so please redirect the univention-translog output to the join.log (in univention-join)
(In reply to Felix Botner from comment #29) > (In reply to Philipp Hahn from comment #28) > > If you prefer "flights in the dark", append "2>/dev/null" or "2>&1 | tee > > log" or "2>&1 | cat". > > this is the way univention-join operates, we have no progress in any of the > join scripts, if you want detailed information, open the join.log > > so please redirect the univention-translog output to the join.log (in > univention-join) [4.3-3] 5609a208d4 Bug #48427 join: Silence progress information. management/univention-join/debian/changelog | 6 ++++++ management/univention-join/univention-join | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) Package: univention-join Version: 10.0.0-29A~4.3.0.201902061728 Branch: ucs_4.3-0 Scope: errata4.3-3 [4.3-3] e8e48fff41 Bug #48427: univention-join 10.0.0-29A~4.3.0.201902061728 doc/errata/staging/univention-join.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
*** Bug 48615 has been marked as a duplicate of this bug. ***
Backup tests failed, the translog database could not be initialized http://jenkins.knut.univention.de:8080/view/Alle/job/UCS-4.3/job/UCS-4.3-3/job/AutotestJoin/
OK - univention-join output (In reply to Felix Botner from comment #32) > Backup tests failed, the translog database could not be initialized > > http://jenkins.knut.univention.de:8080/view/Alle/job/UCS-4.3/job/UCS-4.3-3/ > job/AutotestJoin/ this is a side effect of the "backup join with broken translog database", i am fine if we open a new bug for that and revert the change for now
(In reply to Felix Botner from comment #33) > (In reply to Felix Botner from comment #32) > > Backup tests failed, the translog database could not be initialized > > this is a side effect of the "backup join with broken translog database", i > am fine if we open a new bug for that and revert the change for now No need for a new bug (at least for that): [4.3-3] b7efdb4125 "Bug #48427 join: Move code to force-reset translog DB back management/univention-join/debian/changelog | 6 ++++++ management/univention-join/univention-join | 5 ++++- 2 files changed, 10 insertions(+), 1 deletion(-) Package: univention-join Version: 10.0.0-30A~4.3.0.201902071146 Branch: ucs_4.3-0 Scope: errata4.3-3 TODO: update YAML
(In reply to Philipp Hahn from comment #34) > (In reply to Felix Botner from comment #33) > > (In reply to Felix Botner from comment #32) > > > Backup tests failed, the translog database could not be initialized > > > > this is a side effect of the "backup join with broken translog database", i > > am fine if we open a new bug for that and revert the change for now > > No need for a new bug (at least for that): > > [4.3-3] b7efdb4125 "Bug #48427 join: Move code to force-reset translog DB > back > management/univention-join/debian/changelog | 6 ++++++ > management/univention-join/univention-join | 5 ++++- > 2 files changed, 10 insertions(+), 1 deletion(-) [4.3-3] 1bb088a2c0 Bug #48427 join: Just purge translog management/univention-join/debian/changelog | 6 ++++++ management/univention-join/univention-join | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) Package: univention-join Version: 10.0.0-31A~4.3.0.201902071221 Branch: ucs_4.3-0 Scope: errata4.3-3
* please add something like this print 'checkpoint %s'%configRegistry['ldap/translog/mdb/checkpoint'] print 'envflags %s'%configRegistry['ldap/translog/mdb/envflags'] to the mdb config of the translog database, defaults for envflags should be "nosync", and for checkpint "0 1" * and something like last_tid=$(tail -1 /var/lib/univention-ldap/notify/transaction | awk '{print $1}') if ! /usr/share/univention-directory-notifier/univention-translog \ lookup "$last_id" >/dev/null 2>&1; then /usr/share/univention-directory-notifier/univention-translog \ --lenient import fi to (i guess) /etc/runit/univention-directory-notifier/run
(In reply to Felix Botner from comment #36) > * please add something like this ... > to the mdb config of the translog database, defaults for envflags should be > "nosync", and for checkpint "0 1" > > * and something like ... > to (i guess) /etc/runit/univention-directory-notifier/run [4.3-3] 616e7cfaa6 Bug #48427: udn,ldap: Performance issues .../etc/runit/univention-directory-notifier/run | 14 ++++++- .../univention-directory-notifier/debian/changelog | 6 +++ .../univention-translog | 47 ++++++++++++++++++++-- .../conffiles/etc/ldap/slapd.conf.d/99translog | 2 + management/univention-ldap/debian/changelog | 6 +++ management/univention-ldap/scripts/setup-translog | 1 + 6 files changed, 71 insertions(+), 5 deletions(-) [4.3-3] 36b0c8f0d5 Bug #48427 ldap: Fix translog slapd.conf template .../univention-ldap/conffiles/etc/ldap/slapd.conf.d/99translog | 4 ++-- management/univention-ldap/debian/changelog | 6 ++++++ 2 files changed, 8 insertions(+), 2 deletions(-) Package: univention-directory-notifier Version: 12.0.1-9A~4.3.0.201902071606 Branch: ucs_4.3-0 Scope: errata4.3-3 Package: univention-ldap Version: 14.0.2-39A~4.3.0.201902071608 Version: 14.0.2-40A~4.3.0.201902071627 Branch: ucs_4.3-0 Scope: errata4.3-3 [4.3-3] 4df6d6d652 Bug #48427 udn,ldap,join: YAML doc/errata/staging/univention-directory-notifier.yaml | 2 +- doc/errata/staging/univention-join.yaml | 2 +- doc/errata/staging/univention-ldap.yaml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) [4.3-3] 4fc0f788b9 Bug #43515: univention-ldap 14.0.2-40A~4.3.0.201902071627 doc/errata/staging/univention-ldap.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Looks good, OK - mdb translog config OK - udn run OK - backup join But why last="$(awk 'END{print $1}' "/var/lib/univention-ldap/notify/transaction")" could we not use /var/lib/univention-ldap/last_id? and we need to adjust the limits (again) 0MB 100000 entries 10% (we only want to the last 100000 entries by default), and the limis should be configurable via UCR variables
(In reply to Felix Botner from comment #38) > we need to adjust the limits (again) > > 0MB > 100000 entries > 10% No, as discussed: 10MB, 100.000 entries and 0%
(In reply to Felix Botner from comment #38) > But why > > last="$(awk 'END{print $1}' "/var/lib/univention-ldap/notify/transaction")" > > could we not use /var/lib/univention-ldap/last_id? UDN != UDL (In reply to Sönke Schwardt-Krummrich from comment #39) > (In reply to Felix Botner from comment #38) > > we need to adjust the limits (again) > No, as discussed: 10MB, 100.000 entries and 0% [4.3-3] 9e5c16c95d Bug #48427 udn: Change import limits management/univention-directory-notifier/debian/changelog | 6 ++++++ management/univention-directory-notifier/univention-translog | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) Package: univention-directory-notifier Version: 12.0.1-10A~4.3.0.201902080834 Branch: ucs_4.3-0 Scope: errata4.3-3 [4.3-3] 829c2bc563 Bug #41262: univention-directory-notifier 12.0.1-10A~4.3.0.201902080834 doc/errata/staging/univention-directory-notifier.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Looks goods one (last?) thing, during the update on i386 i got this 5c598e01 bdb_db_open: warning - no DB_CONFIG file found in directory /var/lib/univention-ldap/translog: (2). Expect poor performance for suffix "cn=translog". 2019-02-05 14:22:09.716860188+01:00 (in joinscript_save_current_version) So we better create a DB_CONFIG (maybe just the same as for the bdb "main" database).
(In reply to Felix Botner from comment #41) > one (last?) thing, during the update on i386 i got this > > 5c598e01 bdb_db_open: warning - no DB_CONFIG file found in directory > /var/lib/univention-ldap/translog: (2). ... > So we better create a DB_CONFIG (maybe just the same as for the bdb "main" > database). I simply forgot to register the new UCR template file: [4.3-3] 7201210212 Bug #38327 join: Add DB_CONFIG for translog management/univention-ldap/debian/changelog | 6 ++++++ management/univention-ldap/debian/univention-ldap-server.postinst | 5 ++++- .../debian/univention-ldap-server.univention-config-registry | 5 +++++ management/univention-ldap/scripts/setup-translog | 1 + 4 files changed, 16 insertions(+), 1 deletion(-) Package: univention-ldap Version: 14.0.2-41A~4.3.0.201902081625 Branch: ucs_4.3-0 Scope: errata4.3-3 [4.3-3] 2f8b03ebe2 Bug #43515: univention-ldap 14.0.2-41A~4.3.0.201902081625 doc/errata/staging/univention-ldap.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
OK, that's it OK - univention-directory-listener.yaml OK - univention-directory-notifier.yaml OK - univention-join.yaml OK - univention-ldap.yaml OK univention-management-console-module-diagnostic.yaml
REOPEN: GET_DN still works even when min-version >= 3, so the issue is NOT fixed!
[4.3-3] a28053045b Bug #48427 UDN: Forbid vulnerable GET_DN for VERSION >= 3 management/univention-directory-notifier/debian/changelog | 6 ++++++ management/univention-directory-notifier/src/callback.c | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) Package: univention-directory-notifier Version: 12.0.1-11A~4.3.0.201902131040 Branch: ucs_4.3-0 Scope: errata4.3-3 [4.3-3] 5a4317e1f1 Bug #41262: univention-directory-notifier 12.0.1-11A~4.3.0.201902131040 doc/errata/staging/univention-directory-notifier.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) OK: printf 'Version: 3\nCapabilities: \n\nMSGID: 1\nGET_ID\n\nMSGID: 2\nGET_DN 1\n\n' | nc 10.200.17.50 6669
(In reply to Philipp Hahn from comment #45) > [4.3-3] a28053045b Bug #48427 UDN: Forbid vulnerable GET_DN for VERSION >= 3 > management/univention-directory-notifier/debian/changelog | 6 ++++++ > management/univention-directory-notifier/src/callback.c | 2 +- > 2 files changed, 7 insertions(+), 1 deletion(-) > > Package: univention-directory-notifier > Version: 12.0.1-11A~4.3.0.201902131040 > Branch: ucs_4.3-0 > Scope: errata4.3-3 > > [4.3-3] 5a4317e1f1 Bug #41262: univention-directory-notifier > 12.0.1-11A~4.3.0.201902131040 > doc/errata/staging/univention-directory-notifier.yaml | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > > OK: printf 'Version: 3\nCapabilities: \n\nMSGID: 1\nGET_ID\n\nMSGID: > 2\nGET_DN 1\n\n' | nc 10.200.17.50 6669 OK - univention-directory-notifier OK - yaml
<http://errata.software-univention.de/ucs/4.3/423.html> <http://errata.software-univention.de/ucs/4.3/425.html> <http://errata.software-univention.de/ucs/4.3/426.html> <http://errata.software-univention.de/ucs/4.3/427.html> <http://errata.software-univention.de/ucs/4.3/428.html>
CVE-2019-1010283 <https://nvd.nist.gov/vuln/detail/CVE-2019-1010283>