Bug 44013 - Increase robustness of UMC-Server if LDAP connection gets invalidated
Increase robustness of UMC-Server if LDAP connection gets invalidated
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: UMC (Generic)
UCS 4.2
Other Linux
: P5 normal (vote)
: UCS 4.2
Assigned To: Florian Best
Alexander Kläser
: interim-4
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2017-03-21 19:16 CET by Florian Best
Modified: 2017-04-18 08:03 CEST (History)
0 users

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:
best: Patch_Available+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Florian Best univentionstaff 2017-03-21 19:16:20 CET
We had the case of one UMC-Server which had a broken ldap connection cached. This caused that using the UDM module for every(!) user which authenticated after this hanging connection couldn't use UDM anymore because the DN of the user cannot be searched.

The pseudocode what happens internally:
>>> from univention.management.console.ldap import get_machine_connection
>>> lo, po = get_machine_connection(write=False)
>>> lo.searchDn('uid=Administrator')
['uid=Administrator,cn=users,dc=mydomain,dc=intranet']
>>> lo.unbind()
>>> lo.searchDn('uid=Administrator')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 455, in searchDn
    raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg)
univention.admin.uexceptions.ldapError: LDAP connection invalid
>>> lo.lo.lo.whoami_s()  # :D
Segmentation fault

I am not completely sure if this is a regression in UCS 4.2 but at least it gets worse in UCS 4.2 because now the LDAP connection is cached among sessions to decrease load and prevent against DoS for unauthenticated commands.

The solution is to reset the connection if an LDAP error occur.
Comment 1 Florian Best univentionstaff 2017-03-21 19:20:59 CET
I can't really reproduce this because the reconnecting LDAP connection seems to handle most cases on itself. It happens only if it cannot reconnect.

univention-management-console (9.0.68-1):
r78080 | Bug #44013: prevent hanging or crashing UMC-Server if the ldap connection somehow gets damaged
Comment 2 Florian Best univentionstaff 2017-03-21 19:24:16 CET
Some of the tracebacks which might occur in the logs then are:

18.03.17 04:13:44.321  MAIN        ( ERROR   ) : The LDAP DN for user Administrator could not be found (lo=<univention.admin.uldap.access instance at 0x7fafca50e3f8>)
18.03.17 04:13:44.326  ACL         ( WARN    ) : Error reading credentials from LDAP: Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/univention/management/console/acl.py", line 368, in _read_from_ldap
    userdn = self.lo.searchDn(filter_format('(&(objectClass=person)(uid=%s))', [self.username]), unique=True)[0]
  File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 455, in searchDn
    raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg)
ldapError: Insufficient access


18.03.17 04:21:36.737  AUTH        ( ERROR   ) : Canonicalization of username failed: Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/univention/management/console/auth.py", line 132, in __canonicalize_username
    result = lo.search(filter_format('(&(%s=%s)(objectClass=person))', (attr, username)), attr=['uid'], unique=True)
  File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 439, in search
    raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg)
ldapError: Invalid credentials


  File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/modserver.py", line 178, in _recv
    self.handle(msg)
  File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/modserver.py", line 178, in _recv
    self.handle(msg)
  File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/modserver.py", line 178, in _recv
    self.handle(msg)
  File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/modserver.py", line 178, in _recv
    self.handle(msg)
  File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/modserver.py", line 178, in _recv
    self.handle(msg)
  File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/modserver.py", line 178, in _recv
    self.handle(msg)
  File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/modserver.py", line 286, in handle
    self.__handler.init()
  File "/usr/lib/pymodules/python2.7/univention/management/console/modules/udm/__init__.py", line 151, in init
    raise UserWithoutDN(self._username)
UserWithoutDN: Die LDAP-DN des Benutzers Administrator konnte nicht ermittelt werden.
Comment 3 Alexander Kläser univentionstaff 2017-03-29 23:48:48 CEST
For the QA, I will concentrate on code review, as the problems are too difficult to provoke. So far the UMC server seems to be more robust to these scenarios (looking at it over the past week. A few questions remain.

(In reply to Florian Best from comment #2)
> Some of the tracebacks which might occur in the logs then are:
> 
> 18.03.17 04:13:44.321  MAIN        ( ERROR   ) : The LDAP DN for user
> Administrator could not be found (lo=<univention.admin.uldap.access instance
> at 0x7fafca50e3f8>)
> 18.03.17 04:13:44.326  ACL         ( WARN    ) : Error reading credentials
> from LDAP: Traceback (most recent call last):
>   File "/usr/lib/pymodules/python2.7/univention/management/console/acl.py",
> line 368, in _read_from_ldap
>     userdn =
> self.lo.searchDn(filter_format('(&(objectClass=person)(uid=%s))',
> [self.username]), unique=True)[0]
>   File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 455,
> in searchDn
>     raise univention.admin.uexceptions.ldapError(_err2str(msg),
> original_exception=msg)
> ldapError: Insufficient access

I do not directly see which line would catch this error here.


> 18.03.17 04:21:36.737  AUTH        ( ERROR   ) : Canonicalization of
> username failed: Traceback (most recent call last):
>   File "/usr/lib/pymodules/python2.7/univention/management/console/auth.py",
> line 132, in __canonicalize_username
>     result = lo.search(filter_format('(&(%s=%s)(objectClass=person))',
> (attr, username)), attr=['uid'], unique=True)
>   File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 439,
> in search
>     raise univention.admin.uexceptions.ldapError(_err2str(msg),
> original_exception=msg)
> ldapError: Invalid credentials

You addressed this traceback via 'if lo:' just before this line. Does this work? Would it not be more appropriate to use a "try: ... except ldapError..." clause?


>   File
> "/usr/lib/pymodules/python2.7/univention/management/console/protocol/
> modserver.py", line 178, in _recv
>     self.handle(msg)
>   File
> "/usr/lib/pymodules/python2.7/univention/management/console/protocol/
> modserver.py", line 178, in _recv
>     self.handle(msg)
>   File
> "/usr/lib/pymodules/python2.7/univention/management/console/protocol/
> modserver.py", line 178, in _recv
>     self.handle(msg)
>   File
> "/usr/lib/pymodules/python2.7/univention/management/console/protocol/
> modserver.py", line 178, in _recv
>     self.handle(msg)
>   File
> "/usr/lib/pymodules/python2.7/univention/management/console/protocol/
> modserver.py", line 178, in _recv
>     self.handle(msg)
>   File
> "/usr/lib/pymodules/python2.7/univention/management/console/protocol/
> modserver.py", line 178, in _recv
>     self.handle(msg)
>   File
> "/usr/lib/pymodules/python2.7/univention/management/console/protocol/
> modserver.py", line 286, in handle
>     self.__handler.init()
>   File
> "/usr/lib/pymodules/python2.7/univention/management/console/modules/udm/
> __init__.py", line 151, in init
>     raise UserWithoutDN(self._username)
> UserWithoutDN: Die LDAP-DN des Benutzers Administrator konnte nicht
> ermittelt werden.

OK.
Comment 4 Florian Best univentionstaff 2017-03-30 12:19:37 CEST
(In reply to Alexander Kläser from comment #3)
> For the QA, I will concentrate on code review, as the problems are too
> difficult to provoke. So far the UMC server seems to be more robust to these
> scenarios (looking at it over the past week. A few questions remain.
> 
> (In reply to Florian Best from comment #2)
> > Some of the tracebacks which might occur in the logs then are:
> > 
> > 18.03.17 04:13:44.321  MAIN        ( ERROR   ) : The LDAP DN for user
> > Administrator could not be found (lo=<univention.admin.uldap.access instance
> > at 0x7fafca50e3f8>)
> > 18.03.17 04:13:44.326  ACL         ( WARN    ) : Error reading credentials
> > from LDAP: Traceback (most recent call last):
> >   File "/usr/lib/pymodules/python2.7/univention/management/console/acl.py",
> > line 368, in _read_from_ldap
> >     userdn =
> > self.lo.searchDn(filter_format('(&(objectClass=person)(uid=%s))',
> > [self.username]), unique=True)[0]
> >   File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 455,
> > in searchDn
> >     raise univention.admin.uexceptions.ldapError(_err2str(msg),
> > original_exception=msg)
> > ldapError: Insufficient access
> 
> I do not directly see which line would catch this error here.
svn r78080, the following hunk will catch this:
_read_from_ldap is done in the __init__ of LDAP_ACLs.

 	def _reload_acls(self):
-		self.acls = LDAP_ACLs(self.lo, self._username, ucr['ldap/base'])
+		try:
+			self.acls = LDAP_ACLs(self.lo, self._username, ucr['ldap/base'])
+		except (ldap.LDAPError, udm_errors.ldapError):
+			reset_cache()
+			raise

> > 18.03.17 04:21:36.737  AUTH        ( ERROR   ) : Canonicalization of
> > username failed: Traceback (most recent call last):
> >   File "/usr/lib/pymodules/python2.7/univention/management/console/auth.py",
> > line 132, in __canonicalize_username
> >     result = lo.search(filter_format('(&(%s=%s)(objectClass=person))',
> > (attr, username)), attr=['uid'], unique=True)
> >   File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 439,
> > in search
> >     raise univention.admin.uexceptions.ldapError(_err2str(msg),
> > original_exception=msg)
> > ldapError: Invalid credentials
> 
> You addressed this traceback via 'if lo:' just before this line. Does this
> work? Would it not be more appropriate to use a "try: ... except
> ldapError..." clause?

No, I did not address this via "if lo:". There is a try-except around this:

-		except (ldap.LDAPError, IOError, AttributeError) as exc:
+		except (ldap.LDAPError, udm_errors.ldapError) as exc:
 			# /etc/machine.secret missing or LDAP server not reachable
 			AUTH.warn('Canonicalization of username was not possible: %s' % (exc,))
+			reset_cache()

The "if lo" has been added because I switched from univention.admin.uldap to univention.management.console.ldap which returns None if /etc/machine.secret does not exists instead of raising IOError. Therefore i could remove the IOError thing which is not really obvious why it was there, now it is. Also the AttributeError can be removed as lo is checked and cannot be None anymore.
Comment 5 Florian Best univentionstaff 2017-03-30 12:22:03 CEST
It's also important to remove the IOError and AttributeError from that because otherwise one could DoS attack this if she is able to trigger this, which now resets the cache causing new ldap connection to be established. After this there is a "except:" clause which catches every other exception.
Comment 6 Alexander Kläser univentionstaff 2017-03-30 22:10:51 CEST
OK → VERIFIED
Comment 7 Stefan Gohmann univentionstaff 2017-04-04 18:28:19 CEST
UCS 4.2 has been released:
 https://docs.software-univention.de/release-notes-4.2-0-en.html
 https://docs.software-univention.de/release-notes-4.2-0-de.html

If this error occurs again, please use "Clone This Bug".