Bug 49408 - Opening a lot of container objects is slow
Opening a lot of container objects is slow
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: UDM (Generic)
UCS 4.4
Other Linux
: P5 normal (vote)
: UCS 4.4-0-errata
Assigned To: Dirk Wiesenthal
Florian Best
:
Depends on:
Blocks: 50611
  Show dependency treegraph
 
Reported: 2019-05-02 13:44 CEST by Dirk Wiesenthal
Modified: 2019-12-09 16:59 CET (History)
9 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 2: Will only affect a few installed domains
How will those affected feel about the bug?: 3: A User would likely not purchase the product
User Pain: 0.171
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2019042921000552
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dirk Wiesenthal univentionstaff 2019-05-02 13:44:14 CEST
Opening a container/ou object performs one or two LDAP queries. This is fast, but if you iterate over a list of 1000 ou's and open them all, this takes a noticeable amount of seconds.

This should be cached.

+++ This bug was initially created as a clone of Bug #49329 +++

I created a new VM with UCS, installed UCS@School and created 1000 Schools with:

#!/bin/bash
for i in {1..1000}
do
  /usr/share/ucs-school-import/scripts/create_ou "schule$i"
done

This process took as expected a long time but afterwards the UMC loads for a long time when i'm opening the schools module or LDAP-directory. After several minutes i get a error message "internal server error".

Doing a LDAP-search or similar thing in the terminal works fine.
Comment 1 Dirk Wiesenthal univentionstaff 2019-05-02 13:56:33 CEST
Fixed in
  univention-directory-manager-modules 14.0.12-28A~4.4.0.201905021347

The cache function is now in
  handlers/container/__init__.py
and used in container/cn and container/ou.

The cache is aware of the connection and the ldap base that are passed to the function. This should be irrelevant, yet it feels more correct. The cache in invalidated every 2 seconds (TTL).

No cleanup is performed. If a million connections are passed, a million values are saved. I guess this is not an issue?
Comment 2 Florian Best univentionstaff 2019-05-02 17:32:43 CEST
OK: caching
OK: ttl
OK: memory leak
OK: lookup_filter()
OK: YAML
Comment 3 Arvid Requate univentionstaff 2019-05-15 14:52:27 CEST
<http://errata.software-univention.de/ucs/4.4/102.html>
Comment 4 Florian Best univentionstaff 2019-10-14 15:00:59 CEST
Side effect:
removing many containers twice causes the following exception:

11.10.19 12:08:04.616  ADMIN       ( PROCESS ) : remove: subelement cn=groups,ou=oldschool,l=school,l=dev                      
11.10.19 12:08:04.621  ADMIN       ( ERROR   ) : remove: could not remove 'cn=groups,ou=oldschool,l=school,l=dev': ldapError: No such attribute: modify/delete: univentionGroupsObject: no such value
11.10.19 12:08:04.622  ADMIN       ( ERROR   ) : Traceback (most recent call last):                                               
  File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 1539, in _remove_subtree                         
    subobject.remove(remove_childs=True)                                                                                                                      
  File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 877, in remove
    return self._remove(remove_childs)                                                                    
  File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 1506, in _remove
    self._ldap_pre_remove()                                                                     
  File "/usr/lib/pymodules/python2.7/univention/admin/handlers/container/cn.py", line 255, in _ldap_pre_remove
    self.lo.modify(self.default_dn, changes)                                                      
  File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 902, in modify
    raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg)                       
ldapError: No such attribute: modify/delete: univentionGroupsObject: no such value