Univention Bugzilla – Bug 48390
UDM performance can be improved by optimizing LDAP filters
Last modified: 2019-08-02 17:21:31 CEST
The UDM performance can be greatly improved by optimizing LDAP filters. E.g. the users/user module is using the LDAP filter (&(objectClass=posixAccount)(objectClass=shadowAccount)(objectClass=sambaSamAccount)(objectClass=krb5Principal)(objectClass=krb5KDCEntry)(!(uidNumber=0))(!(uid=*$))(!(univentionObjectFlag=functional))) to find LDAP objects that are suitable for the users/user module. In a UCS@school module, that uses a normal user (teacher) account for LDAP queries, the query costs a huge amount of time to find 2264 users/user objects: 5 runs: 7000ms, 7870ms, 9510ms, 7530ms, 7610ms → avg. 7900ms If the LDAP filter is optimized to use only (univentionObjectType=users/user) the request times were only 50%: 3 runs: 4330ms, 4650ms, 4260ms → avg. 4414ms So, since univentionObjectType is now reliable at all UDM objects, I suggest to simplify the LDAP filters in UDM, to speed up LDAP queries.
In our meeting we were unsure about fixing this (but we would like to): 1. There exists only a diagnose scripts which warns you that your object types are not set. The ldap was not migrated! 2. Are we sure that univentionObjectType is ensured? No customer adds objects via raw ldap? Can we drop support for old UCS versions which might be part of the domain (e.g. UCS 3, for some objects UCS 4.x)? univentionObjectType was added for some DNS and DHCP objects via: * Bug #37504: UCS 4.0 errata 122 * Bug #38073: UCS 4.1 errata 319 Can we break them (because they are out of maintenance)? 3. Is it really necessary to make this change for every object type? Performance relevant should be users, groups, maybe computers!? It adds a unnecessary dependency on univentionObjectType while technically it would be nice if UDM is just a nice tool for official LDAP standards (e.g. DHCP and DNS schema). Would it help to make a (|(univentionObjectType=module/foo)(&(objectClass=old_filters_here))) filter? This combines the fast filter with an OR and executes the slow fallback filter in the other case. Can we ensure slapd evaluates in this order? Does it even increase performance? Or does it make it worse?
Bug #47846: kerberos/kdcentry is not set at all.
univentionObjectType has been added in UCS 3.0: Bug #23111
If you create a user on a UCS 4.2 DC Slave e.g. without samba option the filter univentionObjectType=users/user would find that object. Modifying such a object raises a traceback: # udm users/user modify --dn 'uid=foobar,dc=dev,dc=local' --set street=foo LDAP Error: Object class violation: object class 'sambaSamAccount' requires attribute 'sambaSID' We could of course check in lookup() if this object has all necessary object classes set. But some code uses lookup_filter() without the additional logic in lookup(). The lookup filter is in theory not reliable that it returns well-formed objects anymore.
(In reply to Florian Best from comment #4) > We could of course check in lookup() if this object has all necessary object > classes set. But some code uses lookup_filter() without the additional logic > in lookup(). For example the UMC-UDM module doesn't use lookup() but only lookup_filter() for performance reasons. This would be broken then.
Modifying an user without samba with UCS 4.4 currently yields: The LDAP object could not be saved: LDAP Error Object class violation: object class 'sambaSamAccount' requires attribute 'sambaSID' Opening an user without posix with UCS 4.4 currently yields: Internal server error during "udm/get (users/user)". Request: udm/get (users/user) File "%PY2.7%/notifier/threads.py", line 78, in _run tmp = self._function() File "%PY2.7%/notifier/__init__.py", line 104, in __call__ return self._function( *tmp, **self._kwargs ) File "%PY2.7%/univention/management/console/modules/udm/__init__.py", line 506, in _get obj = module.get(ldap_dn) File "%PY2.7%/univention/management/console/modules/udm/udm_ldap.py", line 86, in _decorated return method(*args, **kwargs) File "%PY2.7%/univention/management/console/ldap.py", line 143, in _decorated result = func(*args, **kwargs) File "%PY2.7%/univention/management/console/modules/udm/udm_ldap.py", line 688, in get UDM_Error(exc).reraise() File "%PY2.7%/univention/management/console/modules/udm/udm_ldap.py", line 678, in get obj.open() File "%PY2.7%/univention/admin/handlers/users/user.py", line 1549, in open self._load_groups(loadGroups) File "%PY2.7%/univention/admin/handlers/users/user.py", line 1585, in _load_groups raise univention.admin.uexceptions.primaryGroup(self.dn) UDM_Error: Default primary group does not exist uid=without_posix,dc=dev,dc=local Opening an object without univentionMail says: The following empty properties were set to default values in the form. These values will be applied when saving. Advanced settings - Mail - Mail quota (MB): 0 Saving it adds the object class. TBD: without kerberos TBD: without person I think with the time these objects will go away. UCS 4.2 is out of maintenance. If we want to go for sure we can filter the result list by using identify() after just calling lookup_filter() in UMC.
Listing of users without posix option would crash the UDM-CLI: # udm users/user list Default primary group does not exist: uid=without_posix,dc=dev,dc=local (In reply to Florian Best from comment #6) > TBD: without kerberos Changing such an object yields: The LDAP object could not be saved: LDAP Error Object class violation: object class 'krb5KDCEntry' requires attribute 'krb5PrincipalName' > TBD: without person Changing such an object yields: The LDAP object could not be saved: LDAP Error Cannot modify object class: structural object class modification from 'person' to 'inetOrgPerson' not allowed
Created attachment 9994 [details] script printing all filters for all modules.py Attaches is a script which prints the actually used ldap filter in a search for each UDM module. This can be used to compare the prior/after change result. python filter_script.py | python -m json.tool > pre.json python filter_script.py | python -m json.tool > post.json diff -u pre.json post.json
Most ldap filters are now created genericly via the default UDM option, and the identify() method uses this generic approach as well. The filter for users/user has been adjusted by setting "use_performant_ldap_search_filter = True" in the UDM handler. For the robustness against "broken" objects we need to fix Bug #45096, so I enabled that fix but only for users/user. univention-directory-manager-modules (14.0.12-26) f1c278af3058 | Bug #48390: Merge branch 'fbest/48390-generic-performant-ldap-search-filters' into 4.4-0 22f93a960ebe | Bug #48390: make sure invalid objects do not crash UDM b4f5dcd5f0b1 | Bug #48390: enable use_performant_ldap_search_filter for users/user b0a692935f77 | Bug #48390: use generic lookup in computers/ b46919b37109 | Bug #48390: unify identify() method a2dc5ec82db5 | Bug #48390: remove also for mail/share c519ef43c4b2 | Bug #48390: remove also for metadata settings 61548f4364b5 | Bug #48390: remove also for policies 1a586f1deb63 | Bug #48390: remove filter which are set in the default option 02c17b8354ea | Bug #48390: make search filters generic 98f1ca860d0c | YAML Bug #48390 univention-management-console-module-udm (9.0.12-4) f1c278af3058 | Bug #48390: Merge branch 'fbest/48390-generic-performant-ldap-search-filters' into 4.4-0 9455188fa1df | Bug #48390: fix superordinate detection 98f1ca860d0c | YAML Bug #48390
Advisory update missing, blocks errata Release [FAIL] changes.valid: Mismatching binary package version: 14.0.12-25A~4.4.0.201904182018 != python-univention-directory-manager-cli 14.0.12-27A~4.4.0.201904301002 from univention-directory-manager-modules 14.0.12-27A~4.4.0.201904301002
OK: lookup filter OK: identify OK: new users/user filter is faster (2000 user. logged in as teacher. old: ~15s new: ~5s) OK: test case OK: yaml -> verified
<http://errata.software-univention.de/ucs/4.4/101.html> <http://errata.software-univention.de/ucs/4.4/102.html>