There are multiple performance issues in the (use of) the function fast_member_add() and fast_member_remove() of groups/group objects. In fact, they are slow_member_add/remove(): 1. The whole UDM group is loaded for every member change operation of users/computers while we would only need to do a LDAP operation to add or remove that user/computer in uniqueMember / memberUid values from the group. 2. After loading the group, a search for all the uniqueMember and memberUid attributes of the group is done again, while we already have those data. 3. When a users/user username is renamed, for each group the user is part of the "memberUid" attribute is completely replaced (`ldap.MOD_REPLACE`) with a list of all values. → Bug #58840 4. Additionally the comparison of DNs is done incorrectly - it just compares DNs lowercase. That doesn't respect special chars in different encoded forms (e.g. + as \+ or \2B. 5. Code style is very bad and bloated. Some dead code exists. What is affected? 1. post removal of a user 2. creation of a user 3. modification of group memberships of user 4. renaming or moving of users 1. post removal of a computer 2. creation of a computer 3. modification of group memberships of computers We should instead: → simply make LDAP operations with the values we know - don't load any more objects → perform only `LDAP_ADD` and `LDAP_DEL` instead of `LDAP_REPLACE` modifications for multi value fields - Switch to the new library `univention.dn.DN` for correct DN comparisons. A sketch demonstrating it in Python expressions: Prior: ```python # in users/user.py def __update_groups(self): old_groups = self.oldinfo.get('groups', []) new_groups = self.info.get('groups', []) # … for group in self.info.get('groups', []): if group and not case_insensitive_in_list(group, old_groups): # gets the whole group and unmap all attributes grpobj = group_mod.object(None, self.lo, self.position, group) grpobj.fast_member_add([self.dn], [new_uid]) # in groups/group.py def fast_member_add(self, memberdnlist, uidlist): ml = [] uids = set() members = set() # another unnecessary search, as these information are already in self.oldattr searchResult = self.lo.authz_connection.get(self.dn, attr=['uniqueMember', 'memberUid']) if searchResult: uids = {x.decode('UTF-8').lower() for x in searchResult.get('memberUid', [])} members = {x.decode('UTF-8').lower() for x in searchResult.get('uniqueMember', [])} add_memberdnlist = [dn for dn in memberdnlist if dn.lower() not in members] if add_memberdnlist: ml.append(('uniqueMember', b'', [x.encode('UTF-8') for x in add_memberdnlist])) if ml: return self.lo.authz_connection.modify(self.dn, ml) ``` After: ```python def __update_groups(self) -> None: old_groups = DN.set(self.oldinfo.get('groups', [])) new_groups = DN.set(self.info.get('groups', [])) for group in (new_groups - old_groups): self.__fast_single_member_add(str(group), self.dn, new_uid) def __fast_single_member_add(self, group, member_dn, member_uid, **kwargs): ml = [ ('memberUid', None, [member_uid.encode('UTF-8')]), ('uniqueMember', None, [member_dn.encode('UTF-8')]), ] self.lo.authz_connection.modify(group, ml, exceptions=True, **kwargs) ``` fast_member_add/remove were introduced in UCS 2.4 Bug #19211 git:6d878c9f0554c761ca97ff028f8dd882a93815cb
The changes have been made for users and computers module. There is still room for more improvement for computer modules and unification of code. The S4-Connector and AD-Connector still use the slow group methods. Maybe they can be improved as well. univention-directory-manager-modules (17.4.5) c9d17224377e | feat(udm): provide fast_single_member_{add,remove} for all modules 51fb6fb63937 | refactor(udm): enhance logging ea399ba182fc | perf(udm): make update of groups for computers fast 032d555118fa | perf(udm-computers): make removal from groups after removing computer object fast 0bca5c3ff1e9 | refactor(udm): use correct DN comparision and simplify code d0006acf0db2 | perf(udm-users): make groups changes fast, by not loooking up group objects 10a315a3e89f | refactor(udm): enhance exception handling for fast_member_remove 3d7a24d7a226 | fix(udm): fix group membership management for DNs containing special chars 4dd783eff07c | refactor(udm): remove dead exception wrapping 3f69b3533cc8 | perf(udm): replace memberUid on rename without LDAP search ce0d1ad22b13 | perf(udm): write only changed memberUid on user rename 34a788953184 | chore(udm): update advisory ucs-test (12.4.3) efc123770c3d | test(udm): test_user_rewrite_member_uid univention-directory-manager-modules.yaml 34a788953184 | chore(udm): update advisory
Some statistics of performance by Felix: - `Domain Users` has 25.000 members - Adding 5000 additional users, old `24m40.893s`, new `12m18.599s` - Renaming a user 10.000 times, old `41m20.134s`, new `26m20.602s`
OK - univention-directory-manager-modules OK - tests OK - advisory
<https://errata.software-univention.de/#/?erratum=5.2x307>