Bug 58899 - Improve performance of group membership updates for users and computers
Summary: Improve performance of group membership updates for users and computers
Status: CLOSED FIXED
Alias: None
Product: UCS
Classification: Unclassified
Component: UDM (Generic)
Version: UCS 5.2
Hardware: Other Linux
: P5 normal
Target Milestone: UCS 5.2-4-errata
Assignee: Florian Best
QA Contact: Felix Botner
URL: https://git.knut.univention.de/univen...
Keywords:
Depends on:
Blocks:
 
Reported: 2025-12-11 13:11 CET by Florian Best
Modified: 2025-12-18 15:59 CET (History)
0 users

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional): Cleanup, Debt Technical, Large environments, UCS Performance
Customer ID:
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Florian Best univentionstaff 2025-12-11 13:11:33 CET
There are multiple performance issues in the (use of) the function fast_member_add() and fast_member_remove() of groups/group objects. In fact, they are slow_member_add/remove():

1. The whole UDM group is loaded for every member change operation of users/computers while we would only need to do a LDAP operation to add or remove that user/computer in uniqueMember / memberUid values from the group.
2. After loading the group, a search for all the uniqueMember and memberUid attributes of the group is done again, while we already have those data. 
3. When a users/user username is renamed, for each group the user is part of the "memberUid" attribute is completely replaced (`ldap.MOD_REPLACE`) with a list of all values. → Bug #58840

4. Additionally the comparison of DNs is done incorrectly - it just compares DNs lowercase. That doesn't respect special chars in different encoded forms (e.g. + as \+ or \2B.
5. Code style is very bad and bloated. Some dead code exists.

What is affected?
1. post removal of a user
2. creation of a user
3. modification of group memberships of user
4. renaming or moving of users
1. post removal of a computer
2. creation of a computer
3. modification of group memberships of computers

We should instead:
→ simply make LDAP operations with the values we know - don't load any more objects
→ perform only `LDAP_ADD` and `LDAP_DEL` instead of `LDAP_REPLACE` modifications for multi value fields
- Switch to the new library `univention.dn.DN` for correct DN comparisons.

A sketch demonstrating it in Python expressions:

Prior:
```python
    # in users/user.py
    def __update_groups(self):
        old_groups = self.oldinfo.get('groups', [])
        new_groups = self.info.get('groups', [])
        # …
        for group in self.info.get('groups', []):
            if group and not case_insensitive_in_list(group, old_groups):
                # gets the whole group and unmap all attributes  
                grpobj = group_mod.object(None, self.lo, self.position, group)
                grpobj.fast_member_add([self.dn], [new_uid])

   # in groups/group.py
    def fast_member_add(self, memberdnlist, uidlist):
        ml = []
        uids = set()
        members = set()

        # another unnecessary search, as these information are already in self.oldattr
        searchResult = self.lo.authz_connection.get(self.dn, attr=['uniqueMember', 'memberUid'])
        if searchResult:
            uids = {x.decode('UTF-8').lower() for x in searchResult.get('memberUid', [])}
            members = {x.decode('UTF-8').lower() for x in searchResult.get('uniqueMember', [])}
        add_memberdnlist = [dn for dn in memberdnlist if dn.lower() not in members]

        if add_memberdnlist:
            ml.append(('uniqueMember', b'', [x.encode('UTF-8') for x in add_memberdnlist]))

        if ml:
            return self.lo.authz_connection.modify(self.dn, ml)
```

After:
```python
     def __update_groups(self) -> None:
        old_groups = DN.set(self.oldinfo.get('groups', []))
        new_groups = DN.set(self.info.get('groups', []))

        for group in (new_groups - old_groups):
            self.__fast_single_member_add(str(group), self.dn, new_uid)

    def __fast_single_member_add(self, group, member_dn, member_uid, **kwargs):
        ml = [
            ('memberUid', None, [member_uid.encode('UTF-8')]),
            ('uniqueMember', None, [member_dn.encode('UTF-8')]),
        ]
        self.lo.authz_connection.modify(group, ml, exceptions=True, **kwargs)
```

fast_member_add/remove were introduced in UCS 2.4 Bug #19211 git:6d878c9f0554c761ca97ff028f8dd882a93815cb
Comment 1 Florian Best univentionstaff 2025-12-11 16:39:27 CET
The changes have been made for users and computers module.
There is still room for more improvement for computer modules and unification of code.
The S4-Connector and AD-Connector still use the slow group methods. Maybe they can be improved as well. 

univention-directory-manager-modules (17.4.5)
c9d17224377e | feat(udm): provide fast_single_member_{add,remove} for all modules
51fb6fb63937 | refactor(udm): enhance logging
ea399ba182fc | perf(udm): make update of groups for computers fast
032d555118fa | perf(udm-computers): make removal from groups after removing computer object fast
0bca5c3ff1e9 | refactor(udm): use correct DN comparision and simplify code
d0006acf0db2 | perf(udm-users): make groups changes fast, by not loooking up group objects
10a315a3e89f | refactor(udm): enhance exception handling for fast_member_remove
3d7a24d7a226 | fix(udm): fix group membership management for DNs containing special chars
4dd783eff07c | refactor(udm): remove dead exception wrapping
3f69b3533cc8 | perf(udm): replace memberUid on rename without LDAP search
ce0d1ad22b13 | perf(udm): write only changed memberUid on user rename
34a788953184 | chore(udm): update advisory

ucs-test (12.4.3)
efc123770c3d | test(udm): test_user_rewrite_member_uid

univention-directory-manager-modules.yaml
34a788953184 | chore(udm): update advisory
Comment 2 Florian Best univentionstaff 2025-12-11 16:41:21 CET
Some statistics of performance by Felix:

- `Domain Users` has 25.000 members
- Adding 5000 additional users, old `24m40.893s`, new `12m18.599s`
- Renaming a user 10.000 times, old `41m20.134s`, new `26m20.602s`
Comment 3 Felix Botner univentionstaff 2025-12-17 10:48:30 CET
OK - univention-directory-manager-modules
OK - tests
OK - advisory