Bug 48427 - Notifier and Listener are using an unencrypted and unauthenticated connection
Notifier and Listener are using an unencrypted and unauthenticated connection
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Notifier (univention-directory-notifier)
UCS 4.3
Other Linux
: P5 major (vote)
: UCS 4.3-3-errata
Assigned To: Philipp Hahn
Felix Botner
:
: 48615 (view as bug list)
Depends on: 39394
Blocks: 48615 48642 48673 48729
  Show dependency treegraph
 
Reported: 2019-01-05 22:42 CET by Sönke Schwardt-Krummrich
Modified: 2019-02-21 19:28 CET (History)
10 users (show)

See Also:
What kind of report is it?: Security Issue
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?: Yes
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional): Security
Max CVSS v3 score: 5.3 (CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N/E:H)


Attachments
1.diff (2.73 KB, patch)
2019-01-21 19:19 CET, Arvid Requate
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Sönke Schwardt-Krummrich univentionstaff 2019-01-05 22:42:50 CET
I just noticed, that the TCP connection between listener and notifier is completely unencrypted and that the notifier does NOT require any form of authentication.

This way, anyone who has access to port 6669 of the affected system/domain has the ability to read out all DNs of the LDAP directory that has been ever used.
Since the DNs also contain sensitive data, the following information may be revealed:
- primary mail addresses (via cn=temporary)
- registered mail domains
- names of IMAP shared folders
- user names
- computer names
- IP networks and used IP addresses
- SIDs and uidNumbers (via cn=temporary)
- installed apps
- ...

I haven't found extremely critical information like passwords. But 
a) this is a problem in regard to data protection and privacy.
b) this simplifies attacks (e.g. username guessing is not required)
c) the data may be used directly for spamming or phishing.

Proof of concept:
(sleep 1s; echo -e "Version: 2\nCapabilities: \n"; sleep 1s; \
 echo -e "MSGID: 1\nGET_ID\n"; sleep 3s) | telnet $HOSTNAME 6669

To get the DN of a known transaction (e.g. 876):
(sleep 1s; echo -e "Version: 2\nCapabilities: \n"; sleep 1s; \
 echo -e "MSGID: 1\nGET_DN 876\n"; sleep 3s) | telnet $HOSTNAME 6669
Comment 1 Arvid Requate univentionstaff 2019-01-07 17:58:38 CET
As discussed:

* Let's add capability level 3 to the notifier, which slightly modifies the current protocol by only sending the change IDs but not any details.

* The notifier writes the details into OpenLDAP below a new DN cn=translog (e.g. cn=123,cn=translog), probably via LDAPI.

* When the listener sees capability level 3 then it fetches the details from these objects in cn=translog. This happens in an authenticated, TLS-protected LDAP connection.

* UCR variables for notifier and listener will allow to set a minimum for the protocol version (to prevent downgrade attacks). In a future UCS version we will raise the default for new installations.

* During update we will transfer a reasonably limited number of last transactions from the file to cn=translog (something in the order of 100k). The limit should be adjustable via UCR. When a client asks for an earlier transaction ID while using capability=3, the notifier should signal that it cannot answer the request.

* We also need a CLI tool to support pruning old transactions from cn=translog.
Comment 2 Philipp Hahn univentionstaff 2019-01-14 16:18:02 CET
CWE-213: Intentional Information Exposure <https://cwe.mitre.org/data/definitions/213.html>
Comment 3 Philipp Hahn univentionstaff 2019-01-18 17:37:18 CET
Fixed in branch: <https://git.knut.univention.de/univention/ucs/commits/phahn/48427-udn>
Request for review.
Comment 4 Arvid Requate univentionstaff 2019-01-21 19:05:47 CET
I've updated the packages from "deb [trusted=yes] http://10.200.18.180/debian/ phahn48427-udn main" on a UCS 4.3-3 errata407 DC Master. updater.log shows:


/var/lib/dpkg/info/univention-directory-notifier.postinst: 117: /var/lib/dpkg/info/univention-directory-notifier.postinst: univention-translog: not found


The path is simply missing. Then I did the steps manually: service udn stop; run univention-translog; service udn start. Then I created a user with udm CLI but no transcation is logged, neither to the translog file nor to the new cn=translog.
Comment 5 Arvid Requate univentionstaff 2019-01-21 19:19:59 CET
Created attachment 9808 [details]
1.diff

With notifier/debug/level=4 nothing gets logged for a udm users/user create.

The notifier hangs in:

root@master10:~# strace -p 15326
strace: Process 15326 attached
epoll_wait(4, 


So I guess that's a collateral damage of commit c9141dc51bb. man epoll_wait says:

       Specifying a timeout of -1
       causes epoll_wait() to block indefinitely, while specifying a timeout
       equal to zero cause epoll_wait() to return immediately, even if no
       events are available.

See attached patch proposals.
Comment 6 Arvid Requate univentionstaff 2019-01-21 19:39:49 CET
Apparently it takes one minute for the notifier to notice the change, I've timed it manually as good as possible:


root@master10:~# tail -f /var/lib/univention-ldap/notify/transaction &
[...]
root@master10:~# date; udm users/user create --set username=user6 --set lastname=last6 --set password=univention
Mi 21. Nov 18:44:55 CET 2018

Object created: uid=user6,dc=ar41i1,dc=qa
root@master10:~# 
root@master10:~# 1370 cn=user6,cn=uid,cn=temporary,cn=univention,dc=ar41i1,dc=qa a
1371 cn=2016,cn=uidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa a
1372 cn=2016,cn=gidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa a
1373 cn=2016,cn=gidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa d
1374 uid=user6,dc=ar41i1,dc=qa a
1375 cn=uidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa m
1376 cn=2016,cn=uidNumber,cn=temporary,cn=univention,dc=ar41i1,dc=qa d
1377 cn=user6,cn=uid,cn=temporary,cn=univention,dc=ar41i1,dc=qa d
1378 uid=user6,dc=ar41i1,dc=qa m
1379 uid=user6,dc=ar41i1,dc=qa m
1380 cn=Domain Users,cn=groups,dc=ar41i1,dc=qa m

root@master10:~# date
Mi 21. Nov 18:45:55 CET 2018
Comment 7 Philipp Hahn univentionstaff 2019-01-22 08:49:54 CET
(In reply to Arvid Requate from comment #4)
> I've updated the packages from "deb [trusted=yes]
> http://10.200.18.180/debian/ phahn48427-udn main" on a UCS 4.3-3 errata407
> DC Master. updater.log shows:
> 
> /var/lib/dpkg/info/univention-directory-notifier.postinst: 117:
> /var/lib/dpkg/info/univention-directory-notifier.postinst:
> univention-translog: not found

[phahn/48427-udn] 61a275c405 fixup! Bug #48427 udn: Load transaction log into LDAP
 .../debian/univention-directory-notifier.postinst                     | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

(In reply to Arvid Requate from comment #5)
> root@master10:~# strace -p 15326
> strace: Process 15326 attached
> epoll_wait(4, 
> 
> 
> So I guess that's a collateral damage of commit c9141dc51bb. man epoll_wait
> says:
> 
>        Specifying a timeout of -1
>        causes epoll_wait() to block indefinitely, while specifying a timeout
>        equal to zero cause epoll_wait() to return immediately, even if no
>        events are available.
> 
> See attached patch proposals.

NO! This would create a busy-wait-loop. The correct fix is
[phahn/48427-udn] 893fb9db5e fixup! Bug #48467 udn: Replace select() with epoll()
 management/univention-directory-notifier/src/network.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


(In reply to Arvid Requate from comment #6)
> Apparently it takes one minute for the notifier to notice the change, I've
> timed it manually as good as possible:

I will have a look.
Comment 8 Arvid Requate univentionstaff 2019-01-23 15:45:53 CET
The current plan is to release the new notifier/listener plus LDAP schema extension in KW 7 (ca. 11.-13.2.). Extensive testing is required for an errata update, e.g. because we currently don't enforce that the DC Master is updated before the replicating servers. Special scenarios like that require extra care. In addition to implementing a solution for this bug Philipp also did a security audit for the notifier code and collected a couple of additional improvement proposals which we will address in a prioritized order.
Comment 9 Felix Botner univentionstaff 2019-01-29 17:49:24 CET
* update on master works but not sure about this output
  (i think at least the set -x should be removed)

univention-ldap-server (14.0.2-35A~~2019-01-29-47721.6e10111f1d71a728a85a9ec241f6d4d27a06af06) wird eingerichtet ...
...
Not updating ldap/shadowbind/ignorefilter
Not updating ldap/maxopenfiles
Not updating ldap/acl/read/anonymous
[ ok ] Stopping slapd (via systemctl): slapd.service.
2019-01-29 13:33:20.181990661+01:00 (in joinscript_init)
Multifile: /etc/ldap/slapd.conf
CRITICAL:__main__:Error in OpenLDAP configuration:
5c504812 WARNING: No dynamic config support for overlay translog.
5c504812 WARNING: No dynamic config support for overlay shadowbind.
5c504812 mdb_db_open: database "cn=translog" cannot be opened: No such file or directory (2). Restore from backup!
5c504812 backend_startup_one (type=mdb, suffix="cn=translog"): bi_db_open failed! (2)
slap_startup failed (test would succeed using the -u switch)

+ trap cleanup EXIT
+ main db
+ local action=db
+ shift
+ setup_db
+ local db
+ dpkg --print-architecture
+ db=data.mdb
+ [ -f /var/lib/univention-ldap/translog/data.mdb ]
+ slapadd -f /etc/ldap/slapd.conf -b cn=translog -l /usr/share/univention-ldap/translog.ldif
_#################### 100.00% eta   none elapsed            none fast!         
Closing DB...
+ :
+ cleanup
+ local rc=0
+ [ 0 -ne 0 ]
+ return 0
2019-01-29 13:33:22.457124412+01:00 (in joinscript_save_current_version)
2019-01-29 13:33:22.470492866+01:00 (in joinscript_init)
[ ok ] Stopping slapd (via systemctl): slapd.service.
[ ok ] Starting slapd (via systemctl): slapd.service.
[ ok ] Restarting univention-directory-listener (via systemctl): univention-directory-listener.service.


* notifier/protocol/version=2 for updates and installation (otherwise th
  installation of new backup system with notifier/protocol/version=3 could
  lead to confusion

* univention-join, translog copy goes like this: 
  slapd start && ssh master slapcat | slapadd
  Better to stop the slapd before slapdd?

* in my setup i have 4000 users, the translog db on the master has 72000 
  (file: 66M) entries, during the update of my backup univention-translog 
  created a translog database with 20000 entries (22M) with no notable delay,
  please consider increasing the minimum (count, size, ...) for the translog
  history

* /usr/share/univention-directory-notifier/univention-translog -h
Traceback (most recent call last):
  File "/usr/share/univention-directory-notifier/univention-translog", line 564, in <module>
    exit(main())
  File "/usr/share/univention-directory-notifier/univention-translog", line 302, in main
    opt = parse_args()
  File "/usr/share/univention-directory-notifier/univention-translog", line 495, in parse_args
    opt = parser.parse_args(args)
  File "/usr/lib/python2.7/argparse.py", line 1701, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1733, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python2.7/argparse.py", line 1939, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/usr/lib/python2.7/argparse.py", line 1879, in consume_optional
    take_action(action, args, option_string)
  File "/usr/lib/python2.7/argparse.py", line 1807, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/lib/python2.7/argparse.py", line 996, in __call__
    parser.print_help()
  File "/usr/lib/python2.7/argparse.py", line 2340, in print_help
    self._print_message(self.format_help(), file)
  File "/usr/lib/python2.7/argparse.py", line 2314, in format_help
    return formatter.format_help()
  File "/usr/lib/python2.7/argparse.py", line 281, in format_help
    help = self._root_section.format_help()
  File "/usr/lib/python2.7/argparse.py", line 211, in format_help
    func(*args)
  File "/usr/lib/python2.7/argparse.py", line 211, in format_help
    func(*args)
  File "/usr/lib/python2.7/argparse.py", line 517, in _format_action
    help_text = self._expand_help(action)
  File "/usr/lib/python2.7/argparse.py", line 603, in _expand_help
    return self._get_help_string(action) % params
KeyError: 'defauls'
Comment 10 Philipp Hahn univentionstaff 2019-01-29 18:20:52 CET
(In reply to Felix Botner from comment #9)
> * update on master works but not sure about this output
>   (i think at least the set -x should be removed)

[phahn/48427-udn] 1ff9b8f695 squash! Bug #48427 ldap: Add transaction log database

> * notifier/protocol/version=2 for updates and installation (otherwise th
>   installation of new backup system with notifier/protocol/version=3 could
>   lead to confusion

[phahn/48427-udn] a8739ba933 squash! Bug #48427 udn: Add protocol version 3

> * univention-join, translog copy goes like this: 
>   slapd start && ssh master slapcat | slapadd
>   Better to stop the slapd before slapdd?

[phahn/48427-udn] 40ab694544 squash! Bug #48427 join: Copy translog.mdb on Backup join

> * in my setup i have 4000 users, the translog db on the master has 72000 
>   (file: 66M) entries, during the update of my backup univention-translog 
>   created a translog database with 20000 entries (22M) with no notable delay,
>   please consider increasing the minimum (count, size, ...) for the translog
>   history

[phahn/48427-udn] d301d0319d squash! Bug #48427 udn: Load transaction log into LDAP

> * /usr/share/univention-directory-notifier/univention-translog -h
> Traceback (most recent call last):
> KeyError: 'defauls'

[phahn/48427-udn] 84b369b5a7 squash! Bug #48427 udn: Load transaction log into LDAP
Comment 11 Felix Botner univentionstaff 2019-01-30 17:59:21 CET
* backup re-join

_#################### 100.00% eta   none elapsed            none fast!
Closing DB...
unknown option -- -
usage: ssh [-1246AaCfGgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
           [-D [bind_address:]port] [-E log_file] [-e escape_char]
           [-F configfile] [-I pkcs11] [-i identity_file]
           [-J [user@]host[:port]] [-L address] [-l login_name] [-m mac_spec]
           [-O ctl_cmd] [-o option] [-p port] [-Q query_option] [-R address]
           [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]]
           [user@]hostname [command]
Configure 04univention-ldap-client.inst

furthermore

@backup-> univention-ssh /tmp/univention  \
  Administrator@master.four.three /usr/sbin/slapcat
5c51d3a8 /etc/ldap/slapd.conf: line 139: invalid path: Permission denied
slapcat: bad configuration file!

seems that the admin user is not allowed to slapcat the master

* /tmp/transaction from internal 32bit server

-> usr/share/univention-directory-notifier/univention-translog --translog /tmp/transaction stat
Traceback (most recent call last):
  File "/usr/share/univention-directory-notifier/univention-translog", line 564, in <module>
    exit(main())
  File "/usr/share/univention-directory-notifier/univention-translog", line 315, in main
    return opt.func(opt, ld) or 0
  File "/usr/share/univention-directory-notifier/univention-translog", line 397, in show_stat
    with Index(opt.translog) as index, Translog(opt.translog, index) as translog:
  File "/usr/share/univention-directory-notifier/univention-translog", line 90, in __enter__
    assert reminder == 0, reminder
AssertionError: 8

maybe this is a 32/64 bit problem (univention-translog was started on 64bit), but

Paketlisten werden gelesen... Fertig
N: Skipping acquire of configured file 'main/binary-i386/Packages' as repository 'http://10.200.18.180/debian phahn48427-udn InRelease' doesn't support architecture 'i386'

So, TODO: i386 tests after merge to master!
Comment 12 Felix Botner univentionstaff 2019-02-04 12:13:32 CET
(In reply to Felix Botner from comment #11)
> 
> * /tmp/transaction from internal 32bit server
> 
> -> usr/share/univention-directory-notifier/univention-translog --translog
> /tmp/transaction stat
> Traceback (most recent call last):
>   File "/usr/share/univention-directory-notifier/univention-translog", line
> 564, in <module>
>     exit(main())
>   File "/usr/share/univention-directory-notifier/univention-translog", line
> 315, in main
>     return opt.func(opt, ld) or 0
>   File "/usr/share/univention-directory-notifier/univention-translog", line
> 397, in show_stat
>     with Index(opt.translog) as index, Translog(opt.translog, index) as
> translog:
>   File "/usr/share/univention-directory-notifier/univention-translog", line
> 90, in __enter__
>     assert reminder == 0, reminder
> AssertionError: 8
> 
> maybe this is a 32/64 bit problem (univention-translog was started on
> 64bit), but
> 
> Paketlisten werden gelesen... Fertig
> N: Skipping acquire of configured file 'main/binary-i386/Packages' as
> repository 'http://10.200.18.180/debian phahn48427-udn InRelease' doesn't
> support architecture 'i386'
> 
> So, TODO: i386 tests after merge to master!


I think we should make univention-translog architecture independent (especially if we fix the join to copy the transaction file instead of slapcat the translog database)

Do we really have to "parse" the index file?
Could we not just get the last n lines from the transaction file?
Comment 13 Philipp Hahn univentionstaff 2019-02-04 15:47:33 CET
(In reply to Felix Botner from comment #11)
> * backup re-join

As discussed with Arvid "univention-join" will use the downloaded transaction.log text file and import it using "univention-translog import" from UDN.
That way no extra file needs to be downloaded.
Another option would have been to change the ACL to allow "Domain Admins" and/or "DC Backup Hosts" read access to "cn=translog".

> unknown option -- -

I have no idea where this comes from, but it should be gone.

> /etc/ldap/slapd.conf: line 139: invalid path: Permission denied

Same as above.

> * /tmp/transaction from internal 32bit server
...
> maybe this is a 32/64 bit problem (univention-translog was started on
> 64bit), but

The index files are architecture dependent.


(In reply to Felix Botner from comment #12)
> I think we should make univention-translog architecture independent
> (especially if we fix the join to copy the transaction file instead of
> slapcat the translog database)

The script is architecture independent, but the files it works on are not.
Similar you cannot copy a LDAP database and expect the slap* tools to work on them.

> Do we really have to "parse" the index file?
> Could we not just get the last n lines from the transaction file?

Yes, one can, but simply stop copying architecture dependent files between incompatible architectures.
One of the reasons I started writing "univention-translog" is to add more sub-commands like "rebuild-index" and "check" to make it easier to work with those architecture dependent things. Another reason was that I already had "fun" with that format during development, but as I was asked to not work on Bug #41687 that code will remain hidden in my branch.
Comment 14 Philipp Hahn univentionstaff 2019-02-05 11:53:37 CET
[4.3-3] d83f374029 Bug #48427: Merge branch 'phahn/48427-udn' into 4.3-3

Package: ucs-test
Version: 8.0.28-261A~4.3.0.201902051137
Branch: ucs_4.3-0
Scope: errata4.3-3

Package: univention-directory-notifier
Version: 12.0.1-4A~4.3.0.201902051143
Branch: ucs_4.3-0
Scope: errata4.3-3

Package: univention-directory-listener
Version: 12.0.0-34A~4.3.0.201902051144
Branch: ucs_4.3-0
Scope: errata4.3-3

Package: univention-join
Version: 10.0.0-26A~4.3.0.201902051146
Branch: ucs_4.3-0
Scope: errata4.3-3

Package: univention-ldap
Version: 14.0.2-36A~4.3.0.201902051148
Branch: ucs_4.3-0
Scope: errata4.3-3

Package: univention-management-console-module-diagnostic
Version: 4.0.1-21A~4.3.0.201902051151
Branch: ucs_4.3-0
Scope: errata4.3-3

[4.3-3] 2e06fe54c5 Bug #48427 udn,udl,ldap,umc-diag: YAML
 doc/errata/staging/univention-directory-listener.yaml                   | 2 +-
 doc/errata/staging/univention-directory-notifier.yaml                   | 2 +-
 doc/errata/staging/univention-join.yaml                                 | 2 +-
 doc/errata/staging/univention-ldap.yaml                                 | 2 +-
 doc/errata/staging/univention-management-console-module-diagnostic.yaml | 2 +-
 5 files changed, 5 insertions(+), 5 deletions(-)
Comment 15 Felix Botner univentionstaff 2019-02-05 14:10:43 CET
FAIL - rejoin (backup) with broken translog database
FAIL - redirect slapadd output in univention-join



OK - Update Tests 

(M)asetr, (B)ackup, (S)lave, M(E)mber
-> update order
M B S E
B S E M
M S E B
S E B M

After every system update, the replication for all systems has been tested (with all notifiers in the domain)

OK - New installation
OK - Rejoin
Comment 16 Felix Botner univentionstaff 2019-02-05 14:19:32 CET
tested with transaction file from out internal master

-> /usr/share/univention-directory-notifier/univention-translog -v -t /opt/transaction   -l import
...

[('objectClass', ['auditObject']), ('reqDN', ['cn=knut,cn=thinclient,cn=policies,dc=knut,dc=univention,dc=de']), ('reqStart', ['20190205141726Z']), ('reqSession', ['20595']), ('reqType', ['\xd9'])]

2019-02-05 14:17:37,761:CRITICAL:ldap_add(reqSession=20595,cn=translog): {'info': 'reqType: value #0 invalid per syntax', 'desc': 'Invalid syntax'}

and aborted after this error.
Comment 17 Felix Botner univentionstaff 2019-02-05 14:31:57 CET
(In reply to Felix Botner from comment #16)
> tested with transaction file from out internal master
> 
> -> /usr/share/univention-directory-notifier/univention-translog -v -t
> /opt/transaction   -l import
> ...
> 
> [('objectClass', ['auditObject']), ('reqDN',
> ['cn=knut,cn=thinclient,cn=policies,dc=knut,dc=univention,dc=de']),
> ('reqStart', ['20190205141726Z']), ('reqSession', ['20595']), ('reqType',
> ['\xd9'])]
> 
> 2019-02-05 14:17:37,761:CRITICAL:ldap_add(reqSession=20595,cn=translog):
> {'info': 'reqType: value #0 invalid per syntax', 'desc': 'Invalid syntax'}
> 
> and aborted after this error.

transaction file corrupt

20595 cn=knut,cn=thinclient,cn=policies,dc=knut,dc=univention,dc=de Ù

we should ignore this error (add a "m" instead, or ignore this transaction)
Comment 18 Felix Botner univentionstaff 2019-02-05 15:56:16 CET
We should reconsider the minima for the "import" command in univention-translog. 

Currently the tool wants to "convert" at least 100MB of the transaction file, this seems too much:

-> wc -l transaction
265472 transaction

-> du -h transaction
25M	transaction

-> time /usr/share/univention-directory-notifier/univention-translog  -t /opt/transaction   -l import
real	3m49,832s

-> du -h /var/lib/univention-ldap/translog/
261M	/var/lib/univention-ldap/translog/

so with the 100MB minimum we would convert a million transaction, which takes 16min during the update and join and would result in a 1G ldap database

Better take the last 100000 transactions and that's it.
Comment 19 Philipp Hahn univentionstaff 2019-02-05 17:23:12 CET
(In reply to Felix Botner from comment #16)
> tested with transaction file from out internal master

which contains invalid UTF-8 byte sequences; univention-translog will print an error message, skip that transaction and continue with the next transaction (when called with option --lenient).  (The "command" we could fix to "m", but not the DN.)


(In reply to Felix Botner from comment #18)
> We should reconsider the minima for the "import" command in
> univention-translog. 

Changed to 10M
Also print a progress report to STDERR when called with a TTY.


[4.3-3] f96530b313 Bug #48427: Merge branch 'phahn/48427-udn' into 4.3-3
[4.3-3] 38b7ec4e01 Bug #48427 udn: Skip invalid transactions
 .../univention-directory-notifier/debian/changelog |  6 +++++
 .../univention-translog                            | 28 +++++++++++++++++-----
 2 files changed, 28 insertions(+), 6 deletions(-)

[4.3-3] d0f8cc581b Bug #48427: Improve setup-translog
 management/univention-join/debian/changelog        |  6 ++
 management/univention-join/univention-join         | 12 +---
 .../01univention-ldap-server-init.inst             |  2 +-
 management/univention-ldap/debian/changelog        |  6 ++
 .../debian/univention-ldap-server.postinst         |  2 +-
 management/univention-ldap/scripts/setup-translog  | 68 +++++++++++++++-------
 6 files changed, 65 insertions(+), 31 deletions(-)

Package: univention-join
Version: 10.0.0-27A~4.3.0.201902051657
Branch: ucs_4.3-0
Scope: errata4.3-3

Package: univention-ldap
Version: 14.0.2-37A~4.3.0.201902051659
Branch: ucs_4.3-0
Scope: errata4.3-3

Package: univention-directory-notifier
Version: 12.0.1-6A~4.3.0.201902051701
Branch: ucs_4.3-0
Scope: errata4.3-3

[4.3-3] 061655d491 Bug #48427 udn,ldap,join: YAML
 doc/errata/staging/univention-directory-notifier.yaml | 2 +-
 doc/errata/staging/univention-join.yaml               | 2 +-
 doc/errata/staging/univention-ldap.yaml               | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
Comment 20 Felix Botner univentionstaff 2019-02-05 18:30:24 CET
why is that?


Package: univention-join
+Breaks: univention-ldap-server (<< 14.0.2-35)
Comment 21 Felix Botner univentionstaff 2019-02-05 18:42:15 CET
(In reply to Felix Botner from comment #20)
> why is that?
> 
> 
> Package: univention-join
> +Breaks: univention-ldap-server (<< 14.0.2-35)

sorry, yes the cn=translog configuration


but the rejoin with an broken translog database still fails

univention-join 10.0.0-27A~4.3.0.20190205165
univention-directory-listener 12.0.0-34A~4.3.0.201
univention-directory-notifier 12.0.1-7A~4.3.0.201902051728

@backup-> univention-join
...
Configure 02univention-directory-notifier.inst             done
Configure 03univention-directory-listener.ins


@backup -> tail -100 /var/log/univention/join.log

05.02.19 18:38:41.021  LISTENER    ( WARN    ) : replication: Can't contact LDAP server: Can not connect LDAP Server, retry in 10 seconds
05.02.19 18:38:51.033  LISTENER    ( WARN    ) : replication: Can't contact LDAP server: Can not connect LDAP Server, retry in 10 seconds
05.02.19 18:39:01.039  LISTENER    ( WARN    ) : replication: Can't contact LDAP server: Can not connect LDAP Server, retry in 10 se


-> mdb_stat /var/lib/univention-ldap/translog/
mdb_env_open failed, error -30793 MDB_INVALID: File is not an LMDB file
Comment 22 Jürn Brodersen univentionstaff 2019-02-06 11:18:48 CET
I got an error message during the update:
'''
Multifile: /etc/ldap/slapd.conf
CRITICAL:__main__:Error in OpenLDAP configuration:
5c5ab162 WARNING: No dynamic config support for overlay translog.
5c5ab162 WARNING: No dynamic config support for overlay shadowbind.
5c5ab162 mdb_db_open: database "cn=translog" cannot be opened: No such file or directory (2). Restore from backup!
5c5ab162 backend_startup_one (type=mdb, suffix="cn=translog"): bi_db_open failed! (2)
slap_startup failed (test would succeed using the -u switch)
'''

But after the update slapd was running and cn=translog looked fine. So I guess this can be ignored?
Comment 23 Jürn Brodersen univentionstaff 2019-02-06 11:19:51 CET
(In reply to Jürn Brodersen from comment #22)
> I got an error message during the update:
> '''
> Multifile: /etc/ldap/slapd.conf
> CRITICAL:__main__:Error in OpenLDAP configuration:
> 5c5ab162 WARNING: No dynamic config support for overlay translog.
> 5c5ab162 WARNING: No dynamic config support for overlay shadowbind.
> 5c5ab162 mdb_db_open: database "cn=translog" cannot be opened: No such file
> or directory (2). Restore from backup!
> 5c5ab162 backend_startup_one (type=mdb, suffix="cn=translog"): bi_db_open
> failed! (2)
> slap_startup failed (test would succeed using the -u switch)
> '''
> 
> But after the update slapd was running and cn=translog looked fine. So I
> guess this can be ignored?

Sorry already reported in comment 9
Comment 24 Felix Botner univentionstaff 2019-02-06 12:41:30 CET
please add this

+++ b/management/univention-directory-notifier/univention-translog
@@ -390,7 +390,7 @@ def ldapi(opt):
     log.debug("ldap_set_option(PROTOCOL_VERSION)")
     ldap.set_option(ldap.OPT_PROTOCOL_VERSION, ldap.VERSION3)
     log.debug("ldap_initialize(%s)", opt.ldap)
-    ld = ReconnectLDAPObject(opt.ldap, trace_level=max(0, opt.verbose - 4))
+    ld = ReconnectLDAPObject(opt.ldap, trace_level=max(0, opt.verbose - 4), retry_max=3, retry_delay=10.0)
     log.debug("ldap_bind()")
     ld.sasl_external_bind_s()


currently the python-ldap reconnect object is not configured and a simple restart of the slapd kills the script (and due to Bug #48530, this is what happens during the update)
Comment 25 Philipp Hahn univentionstaff 2019-02-06 13:15:34 CET
(In reply to Felix Botner from comment #24)
> please add this
...> currently the python-ldap reconnect object is not configured and a simple
> restart of the slapd kills the script (and due to Bug #48530, this is what
> happens during the update)

[4.3-3] 57604d5694 Bug #48427 udn: Fine-tune ReconnectLDAPObject()
 management/univention-directory-notifier/univention-translog | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
[4.3-3] a7a92ad1db Bug #48427 udn: Fix Python format strings
 management/univention-directory-notifier/debian/changelog    | 6 ++++++
 management/univention-directory-notifier/univention-translog | 6 +++---
 2 files changed, 9 insertions(+), 3 deletions(-)


(In reply to Felix Botner from comment #21)
> but the rejoin with an broken translog database still fails

This should be fixed by this commit, but my VMs currently fail to test it:

[4.3-3] 0dbc0f283b Bug #48427 join: Move code to force-reset translog DB.
 management/univention-join/debian/changelog | 6 ++++++
 management/univention-join/univention-join  | 8 +++++---
 2 files changed, 11 insertions(+), 3 deletions(-)


Package: univention-join
Version: 10.0.0-28A~4.3.0.201902061301
Branch: ucs_4.3-0
Scope: errata4.3-3

Package: univention-directory-notifier
Version: 12.0.1-8A~4.3.0.201902061304
Branch: ucs_4.3-0
Scope: errata4.3-3

[4.3-3] f63c6c4c70 Bug #48427 udl,join: YAML
 doc/errata/staging/univention-directory-notifier.yaml | 2 +-
 doc/errata/staging/univention-join.yaml               | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
Comment 26 Felix Botner univentionstaff 2019-02-06 15:01:51 CET
please remove (log to join.log) the univention-translog output in univention-join

-> univention-join
...
Configure 01univention-ldap-server-init.inst               done
Configure 02univention-directory-notifier.inst             done
Configure 03univention-directory-listener.inst             done
processed 54124 of 60597 [89.3%] in 60.0s, 6473 remaining in 7.2s
Comment 27 Felix Botner univentionstaff 2019-02-06 15:06:54 CET
OK - join with broken translog database
OK - logging in univention-translog
OK - ldap reconnect in univention-translog
Comment 28 Philipp Hahn univentionstaff 2019-02-06 15:17:36 CET
(In reply to Felix Botner from comment #26)
> please remove (log to join.log) the univention-translog output in
> univention-join
> 
> -> univention-join
> ...
> processed 54124 of 60597 [89.3%] in 60.0s, 6473 remaining in 7.2s

This is the progress report from univention-translog. You only get it if STDERR is a valid terminal which indicates, that the user invoked it by hand (and not via some script or with some pipe redirection). I consider it a major pain to get NO progress information while somethings happens in the background, especially if you don't know how long it will take. We already scratched our heads that it takes longer even on modern PCs with SSDs than with our VMs using cache="unsafe", so think about users with slow PCs or many transactions to import. They will appreciate that progress information.

If you prefer "flights in the dark", append "2>/dev/null" or "2>&1 | tee log" or "2>&1 | cat".
Comment 29 Felix Botner univentionstaff 2019-02-06 15:58:15 CET
(In reply to Philipp Hahn from comment #28)

> 
> If you prefer "flights in the dark", append "2>/dev/null" or "2>&1 | tee
> log" or "2>&1 | cat".

this is the way univention-join operates, we have no progress in any of the join scripts, if you want detailed information, open the join.log

so please redirect the univention-translog output to the join.log (in univention-join)
Comment 30 Philipp Hahn univentionstaff 2019-02-06 17:29:50 CET
(In reply to Felix Botner from comment #29)
> (In reply to Philipp Hahn from comment #28)
> > If you prefer "flights in the dark", append "2>/dev/null" or "2>&1 | tee
> > log" or "2>&1 | cat".
> 
> this is the way univention-join operates, we have no progress in any of the
> join scripts, if you want detailed information, open the join.log
> 
> so please redirect the univention-translog output to the join.log (in
> univention-join)

[4.3-3] 5609a208d4 Bug #48427 join: Silence progress information.
 management/univention-join/debian/changelog | 6 ++++++
 management/univention-join/univention-join  | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

Package: univention-join
Version: 10.0.0-29A~4.3.0.201902061728
Branch: ucs_4.3-0
Scope: errata4.3-3

[4.3-3] e8e48fff41 Bug #48427: univention-join 10.0.0-29A~4.3.0.201902061728
 doc/errata/staging/univention-join.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 31 Philipp Hahn univentionstaff 2019-02-07 09:30:53 CET
*** Bug 48615 has been marked as a duplicate of this bug. ***
Comment 32 Felix Botner univentionstaff 2019-02-07 10:46:13 CET
Backup tests failed, the translog database could not be initialized

http://jenkins.knut.univention.de:8080/view/Alle/job/UCS-4.3/job/UCS-4.3-3/job/AutotestJoin/
Comment 33 Felix Botner univentionstaff 2019-02-07 11:40:06 CET
OK - univention-join output


(In reply to Felix Botner from comment #32)
> Backup tests failed, the translog database could not be initialized
> 
> http://jenkins.knut.univention.de:8080/view/Alle/job/UCS-4.3/job/UCS-4.3-3/
> job/AutotestJoin/

this is a side effect of the "backup join with broken translog database", i am fine if we open a new bug for that and revert the change for now
Comment 34 Philipp Hahn univentionstaff 2019-02-07 11:49:28 CET
(In reply to Felix Botner from comment #33)
> (In reply to Felix Botner from comment #32)
> > Backup tests failed, the translog database could not be initialized
> 
> this is a side effect of the "backup join with broken translog database", i
> am fine if we open a new bug for that and revert the change for now

No need for a new bug (at least for that):

[4.3-3] b7efdb4125 "Bug #48427 join: Move code to force-reset translog DB back
 management/univention-join/debian/changelog | 6 ++++++
 management/univention-join/univention-join  | 5 ++++-
 2 files changed, 10 insertions(+), 1 deletion(-)

Package: univention-join
Version: 10.0.0-30A~4.3.0.201902071146
Branch: ucs_4.3-0
Scope: errata4.3-3

TODO: update YAML
Comment 35 Philipp Hahn univentionstaff 2019-02-07 12:23:33 CET
(In reply to Philipp Hahn from comment #34)
> (In reply to Felix Botner from comment #33)
> > (In reply to Felix Botner from comment #32)
> > > Backup tests failed, the translog database could not be initialized
> > 
> > this is a side effect of the "backup join with broken translog database", i
> > am fine if we open a new bug for that and revert the change for now
> 
> No need for a new bug (at least for that):
> 
> [4.3-3] b7efdb4125 "Bug #48427 join: Move code to force-reset translog DB
> back
>  management/univention-join/debian/changelog | 6 ++++++
>  management/univention-join/univention-join  | 5 ++++-
>  2 files changed, 10 insertions(+), 1 deletion(-)

[4.3-3] 1bb088a2c0 Bug #48427 join: Just purge translog
 management/univention-join/debian/changelog | 6 ++++++
 management/univention-join/univention-join  | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

Package: univention-join
Version: 10.0.0-31A~4.3.0.201902071221
Branch: ucs_4.3-0
Scope: errata4.3-3
Comment 36 Felix Botner univentionstaff 2019-02-07 14:02:27 CET
* please add something like this

        print 'checkpoint %s'%configRegistry['ldap/translog/mdb/checkpoint']
        print 'envflags %s'%configRegistry['ldap/translog/mdb/envflags']

to the mdb config of the translog database, defaults for envflags should be "nosync", and for checkpint "0 1"

* and something like

last_tid=$(tail -1 /var/lib/univention-ldap/notify/transaction | awk '{print
$1}')

if ! /usr/share/univention-directory-notifier/univention-translog \
lookup "$last_id" >/dev/null 2>&1; then
/usr/share/univention-directory-notifier/univention-translog \
--lenient import
fi

to (i guess) /etc/runit/univention-directory-notifier/run
Comment 37 Philipp Hahn univentionstaff 2019-02-07 16:31:30 CET
(In reply to Felix Botner from comment #36)
> * please add something like this
...
> to the mdb config of the translog database, defaults for envflags should be
> "nosync", and for checkpint "0 1"
> 
> * and something like
...
> to (i guess) /etc/runit/univention-directory-notifier/run

[4.3-3] 616e7cfaa6 Bug #48427: udn,ldap: Performance issues
 .../etc/runit/univention-directory-notifier/run    | 14 ++++++-
 .../univention-directory-notifier/debian/changelog |  6 +++
 .../univention-translog                            | 47 ++++++++++++++++++++--
 .../conffiles/etc/ldap/slapd.conf.d/99translog     |  2 +
 management/univention-ldap/debian/changelog        |  6 +++
 management/univention-ldap/scripts/setup-translog  |  1 +
 6 files changed, 71 insertions(+), 5 deletions(-)
[4.3-3] 36b0c8f0d5 Bug #48427 ldap: Fix translog slapd.conf template
 .../univention-ldap/conffiles/etc/ldap/slapd.conf.d/99translog      | 4 ++--
 management/univention-ldap/debian/changelog                         | 6 ++++++
 2 files changed, 8 insertions(+), 2 deletions(-)

Package: univention-directory-notifier
Version: 12.0.1-9A~4.3.0.201902071606
Branch: ucs_4.3-0
Scope: errata4.3-3

Package: univention-ldap
Version: 14.0.2-39A~4.3.0.201902071608
Version: 14.0.2-40A~4.3.0.201902071627
Branch: ucs_4.3-0
Scope: errata4.3-3

[4.3-3] 4df6d6d652 Bug #48427 udn,ldap,join: YAML
 doc/errata/staging/univention-directory-notifier.yaml | 2 +-
 doc/errata/staging/univention-join.yaml               | 2 +-
 doc/errata/staging/univention-ldap.yaml               | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
[4.3-3] 4fc0f788b9 Bug #43515: univention-ldap 14.0.2-40A~4.3.0.201902071627
 doc/errata/staging/univention-ldap.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 38 Felix Botner univentionstaff 2019-02-07 17:43:55 CET
Looks good, 

OK - mdb translog config
OK - udn run
OK - backup join

But why

 last="$(awk 'END{print $1}' "/var/lib/univention-ldap/notify/transaction")"

could we not use /var/lib/univention-ldap/last_id?


and

we need to adjust the limits (again)

 0MB
 100000 entries
 10%

(we only want to the last 100000 entries by default), and the limis should be configurable via UCR variables
Comment 39 Sönke Schwardt-Krummrich univentionstaff 2019-02-07 17:56:10 CET
(In reply to Felix Botner from comment #38)
> we need to adjust the limits (again)
> 
>  0MB
>  100000 entries
>  10%

No, as discussed: 10MB, 100.000 entries and 0%
Comment 40 Philipp Hahn univentionstaff 2019-02-08 08:37:21 CET
(In reply to Felix Botner from comment #38)
> But why
> 
>  last="$(awk 'END{print $1}' "/var/lib/univention-ldap/notify/transaction")"
> 
> could we not use /var/lib/univention-ldap/last_id?

UDN != UDL


(In reply to Sönke Schwardt-Krummrich from comment #39)
> (In reply to Felix Botner from comment #38)
> > we need to adjust the limits (again)
> No, as discussed: 10MB, 100.000 entries and 0%

[4.3-3] 9e5c16c95d Bug #48427 udn: Change import limits
 management/univention-directory-notifier/debian/changelog    | 6 ++++++
 management/univention-directory-notifier/univention-translog | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

Package: univention-directory-notifier
Version: 12.0.1-10A~4.3.0.201902080834
Branch: ucs_4.3-0
Scope: errata4.3-3

[4.3-3] 829c2bc563 Bug #41262: univention-directory-notifier 12.0.1-10A~4.3.0.201902080834
 doc/errata/staging/univention-directory-notifier.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 41 Felix Botner univentionstaff 2019-02-08 11:41:18 CET
Looks goods

one (last?) thing, during the update on i386 i got this

5c598e01 bdb_db_open: warning - no DB_CONFIG file found in directory /var/lib/univention-ldap/translog: (2).
Expect poor performance for suffix "cn=translog".
2019-02-05 14:22:09.716860188+01:00 (in joinscript_save_current_version)

So we better create a DB_CONFIG (maybe just the same as for the bdb "main" database).
Comment 42 Philipp Hahn univentionstaff 2019-02-08 16:27:43 CET
(In reply to Felix Botner from comment #41)
> one (last?) thing, during the update on i386 i got this
> 
> 5c598e01 bdb_db_open: warning - no DB_CONFIG file found in directory
> /var/lib/univention-ldap/translog: (2).
...
> So we better create a DB_CONFIG (maybe just the same as for the bdb "main"
> database).

I simply forgot to register the new UCR template file:
[4.3-3] 7201210212 Bug #38327 join: Add DB_CONFIG for translog
 management/univention-ldap/debian/changelog                         | 6 ++++++
 management/univention-ldap/debian/univention-ldap-server.postinst   | 5 ++++-
 .../debian/univention-ldap-server.univention-config-registry        | 5 +++++
 management/univention-ldap/scripts/setup-translog                   | 1 +
 4 files changed, 16 insertions(+), 1 deletion(-)

Package: univention-ldap
Version: 14.0.2-41A~4.3.0.201902081625
Branch: ucs_4.3-0
Scope: errata4.3-3

[4.3-3] 2f8b03ebe2 Bug #43515: univention-ldap 14.0.2-41A~4.3.0.201902081625
 doc/errata/staging/univention-ldap.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 43 Felix Botner univentionstaff 2019-02-11 11:02:55 CET
OK, that's it


OK - univention-directory-listener.yaml
OK - univention-directory-notifier.yaml
OK - univention-join.yaml
OK - univention-ldap.yaml
OK  univention-management-console-module-diagnostic.yaml
Comment 44 Philipp Hahn univentionstaff 2019-02-13 10:16:30 CET
REOPEN: GET_DN still works even when min-version >= 3, so the issue is NOT fixed!
Comment 45 Philipp Hahn univentionstaff 2019-02-13 10:54:08 CET
[4.3-3] a28053045b Bug #48427 UDN: Forbid vulnerable GET_DN for VERSION >= 3
 management/univention-directory-notifier/debian/changelog | 6 ++++++
 management/univention-directory-notifier/src/callback.c   | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

Package: univention-directory-notifier
Version: 12.0.1-11A~4.3.0.201902131040
Branch: ucs_4.3-0
Scope: errata4.3-3

[4.3-3] 5a4317e1f1 Bug #41262: univention-directory-notifier 12.0.1-11A~4.3.0.201902131040
 doc/errata/staging/univention-directory-notifier.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


OK: printf 'Version: 3\nCapabilities: \n\nMSGID: 1\nGET_ID\n\nMSGID: 2\nGET_DN 1\n\n' | nc 10.200.17.50 6669
Comment 46 Felix Botner univentionstaff 2019-02-13 12:48:04 CET
(In reply to Philipp Hahn from comment #45)
> [4.3-3] a28053045b Bug #48427 UDN: Forbid vulnerable GET_DN for VERSION >= 3
>  management/univention-directory-notifier/debian/changelog | 6 ++++++
>  management/univention-directory-notifier/src/callback.c   | 2 +-
>  2 files changed, 7 insertions(+), 1 deletion(-)
> 
> Package: univention-directory-notifier
> Version: 12.0.1-11A~4.3.0.201902131040
> Branch: ucs_4.3-0
> Scope: errata4.3-3
> 
> [4.3-3] 5a4317e1f1 Bug #41262: univention-directory-notifier
> 12.0.1-11A~4.3.0.201902131040
>  doc/errata/staging/univention-directory-notifier.yaml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> 
> OK: printf 'Version: 3\nCapabilities: \n\nMSGID: 1\nGET_ID\n\nMSGID:
> 2\nGET_DN 1\n\n' | nc 10.200.17.50 6669

OK - univention-directory-notifier
OK - yaml