Univention Bugzilla – Bug 44013
Increase robustness of UMC-Server if LDAP connection gets invalidated
Last modified: 2017-04-18 08:03:59 CEST
We had the case of one UMC-Server which had a broken ldap connection cached. This caused that using the UDM module for every(!) user which authenticated after this hanging connection couldn't use UDM anymore because the DN of the user cannot be searched. The pseudocode what happens internally: >>> from univention.management.console.ldap import get_machine_connection >>> lo, po = get_machine_connection(write=False) >>> lo.searchDn('uid=Administrator') ['uid=Administrator,cn=users,dc=mydomain,dc=intranet'] >>> lo.unbind() >>> lo.searchDn('uid=Administrator') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 455, in searchDn raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg) univention.admin.uexceptions.ldapError: LDAP connection invalid >>> lo.lo.lo.whoami_s() # :D Segmentation fault I am not completely sure if this is a regression in UCS 4.2 but at least it gets worse in UCS 4.2 because now the LDAP connection is cached among sessions to decrease load and prevent against DoS for unauthenticated commands. The solution is to reset the connection if an LDAP error occur.
I can't really reproduce this because the reconnecting LDAP connection seems to handle most cases on itself. It happens only if it cannot reconnect. univention-management-console (9.0.68-1): r78080 | Bug #44013: prevent hanging or crashing UMC-Server if the ldap connection somehow gets damaged
Some of the tracebacks which might occur in the logs then are: 18.03.17 04:13:44.321 MAIN ( ERROR ) : The LDAP DN for user Administrator could not be found (lo=<univention.admin.uldap.access instance at 0x7fafca50e3f8>) 18.03.17 04:13:44.326 ACL ( WARN ) : Error reading credentials from LDAP: Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/univention/management/console/acl.py", line 368, in _read_from_ldap userdn = self.lo.searchDn(filter_format('(&(objectClass=person)(uid=%s))', [self.username]), unique=True)[0] File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 455, in searchDn raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg) ldapError: Insufficient access 18.03.17 04:21:36.737 AUTH ( ERROR ) : Canonicalization of username failed: Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/univention/management/console/auth.py", line 132, in __canonicalize_username result = lo.search(filter_format('(&(%s=%s)(objectClass=person))', (attr, username)), attr=['uid'], unique=True) File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 439, in search raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg) ldapError: Invalid credentials File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/modserver.py", line 178, in _recv self.handle(msg) File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/modserver.py", line 178, in _recv self.handle(msg) File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/modserver.py", line 178, in _recv self.handle(msg) File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/modserver.py", line 178, in _recv self.handle(msg) File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/modserver.py", line 178, in _recv self.handle(msg) File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/modserver.py", line 178, in _recv self.handle(msg) File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/modserver.py", line 286, in handle self.__handler.init() File "/usr/lib/pymodules/python2.7/univention/management/console/modules/udm/__init__.py", line 151, in init raise UserWithoutDN(self._username) UserWithoutDN: Die LDAP-DN des Benutzers Administrator konnte nicht ermittelt werden.
For the QA, I will concentrate on code review, as the problems are too difficult to provoke. So far the UMC server seems to be more robust to these scenarios (looking at it over the past week. A few questions remain. (In reply to Florian Best from comment #2) > Some of the tracebacks which might occur in the logs then are: > > 18.03.17 04:13:44.321 MAIN ( ERROR ) : The LDAP DN for user > Administrator could not be found (lo=<univention.admin.uldap.access instance > at 0x7fafca50e3f8>) > 18.03.17 04:13:44.326 ACL ( WARN ) : Error reading credentials > from LDAP: Traceback (most recent call last): > File "/usr/lib/pymodules/python2.7/univention/management/console/acl.py", > line 368, in _read_from_ldap > userdn = > self.lo.searchDn(filter_format('(&(objectClass=person)(uid=%s))', > [self.username]), unique=True)[0] > File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 455, > in searchDn > raise univention.admin.uexceptions.ldapError(_err2str(msg), > original_exception=msg) > ldapError: Insufficient access I do not directly see which line would catch this error here. > 18.03.17 04:21:36.737 AUTH ( ERROR ) : Canonicalization of > username failed: Traceback (most recent call last): > File "/usr/lib/pymodules/python2.7/univention/management/console/auth.py", > line 132, in __canonicalize_username > result = lo.search(filter_format('(&(%s=%s)(objectClass=person))', > (attr, username)), attr=['uid'], unique=True) > File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 439, > in search > raise univention.admin.uexceptions.ldapError(_err2str(msg), > original_exception=msg) > ldapError: Invalid credentials You addressed this traceback via 'if lo:' just before this line. Does this work? Would it not be more appropriate to use a "try: ... except ldapError..." clause? > File > "/usr/lib/pymodules/python2.7/univention/management/console/protocol/ > modserver.py", line 178, in _recv > self.handle(msg) > File > "/usr/lib/pymodules/python2.7/univention/management/console/protocol/ > modserver.py", line 178, in _recv > self.handle(msg) > File > "/usr/lib/pymodules/python2.7/univention/management/console/protocol/ > modserver.py", line 178, in _recv > self.handle(msg) > File > "/usr/lib/pymodules/python2.7/univention/management/console/protocol/ > modserver.py", line 178, in _recv > self.handle(msg) > File > "/usr/lib/pymodules/python2.7/univention/management/console/protocol/ > modserver.py", line 178, in _recv > self.handle(msg) > File > "/usr/lib/pymodules/python2.7/univention/management/console/protocol/ > modserver.py", line 178, in _recv > self.handle(msg) > File > "/usr/lib/pymodules/python2.7/univention/management/console/protocol/ > modserver.py", line 286, in handle > self.__handler.init() > File > "/usr/lib/pymodules/python2.7/univention/management/console/modules/udm/ > __init__.py", line 151, in init > raise UserWithoutDN(self._username) > UserWithoutDN: Die LDAP-DN des Benutzers Administrator konnte nicht > ermittelt werden. OK.
(In reply to Alexander Kläser from comment #3) > For the QA, I will concentrate on code review, as the problems are too > difficult to provoke. So far the UMC server seems to be more robust to these > scenarios (looking at it over the past week. A few questions remain. > > (In reply to Florian Best from comment #2) > > Some of the tracebacks which might occur in the logs then are: > > > > 18.03.17 04:13:44.321 MAIN ( ERROR ) : The LDAP DN for user > > Administrator could not be found (lo=<univention.admin.uldap.access instance > > at 0x7fafca50e3f8>) > > 18.03.17 04:13:44.326 ACL ( WARN ) : Error reading credentials > > from LDAP: Traceback (most recent call last): > > File "/usr/lib/pymodules/python2.7/univention/management/console/acl.py", > > line 368, in _read_from_ldap > > userdn = > > self.lo.searchDn(filter_format('(&(objectClass=person)(uid=%s))', > > [self.username]), unique=True)[0] > > File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 455, > > in searchDn > > raise univention.admin.uexceptions.ldapError(_err2str(msg), > > original_exception=msg) > > ldapError: Insufficient access > > I do not directly see which line would catch this error here. svn r78080, the following hunk will catch this: _read_from_ldap is done in the __init__ of LDAP_ACLs. def _reload_acls(self): - self.acls = LDAP_ACLs(self.lo, self._username, ucr['ldap/base']) + try: + self.acls = LDAP_ACLs(self.lo, self._username, ucr['ldap/base']) + except (ldap.LDAPError, udm_errors.ldapError): + reset_cache() + raise > > 18.03.17 04:21:36.737 AUTH ( ERROR ) : Canonicalization of > > username failed: Traceback (most recent call last): > > File "/usr/lib/pymodules/python2.7/univention/management/console/auth.py", > > line 132, in __canonicalize_username > > result = lo.search(filter_format('(&(%s=%s)(objectClass=person))', > > (attr, username)), attr=['uid'], unique=True) > > File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 439, > > in search > > raise univention.admin.uexceptions.ldapError(_err2str(msg), > > original_exception=msg) > > ldapError: Invalid credentials > > You addressed this traceback via 'if lo:' just before this line. Does this > work? Would it not be more appropriate to use a "try: ... except > ldapError..." clause? No, I did not address this via "if lo:". There is a try-except around this: - except (ldap.LDAPError, IOError, AttributeError) as exc: + except (ldap.LDAPError, udm_errors.ldapError) as exc: # /etc/machine.secret missing or LDAP server not reachable AUTH.warn('Canonicalization of username was not possible: %s' % (exc,)) + reset_cache() The "if lo" has been added because I switched from univention.admin.uldap to univention.management.console.ldap which returns None if /etc/machine.secret does not exists instead of raising IOError. Therefore i could remove the IOError thing which is not really obvious why it was there, now it is. Also the AttributeError can be removed as lo is checked and cannot be None anymore.
It's also important to remove the IOError and AttributeError from that because otherwise one could DoS attack this if she is able to trigger this, which now resets the cache causing new ldap connection to be established. After this there is a "except:" clause which catches every other exception.
OK → VERIFIED
UCS 4.2 has been released: https://docs.software-univention.de/release-notes-4.2-0-en.html https://docs.software-univention.de/release-notes-4.2-0-de.html If this error occurs again, please use "Clone This Bug".