Univention Bugzilla – Bug 49862
The initialization of the listener module portal_groups takes too long
Last modified: 2020-08-21 16:44:39 CEST
The initialization of the listener module portal_groups takes up to 12 hours in large environments (36000 groups).
The listener builds its cache group by group the first time. The cache is written 36,000 times instead of calculating everything once at the end.
We have to optimize that.
In this special environment we have the workaround, that this module can be deactivated by setting the ucr varible listener/module/<name>/deactivate
This might not work on other environments.
I small update. The customer has concerns about the Listener module to cause a lot of trouble during the next import of the new user data at the end of the school year. Thus several Dozens of thousands of groups can be modified hundreds of times, so that the Listener module could also stop other modules (S4-Connector) here.
That would be a big problem. If these concerns are justified, the customer would therefore like a short-term performance optimization for the module if possible.
Maybe it's enough to deactivate the module with the ucr variable again?
Is there a reason why the cache is rebuilt with every group change and not just once in postrun()?
Even for environments not that large, this is a problem, especiall since it takes place during an upgrade and extends the downtime and maintenance window massively.
6500 groups on a 8 Core, 32 GB RAM, UCS Backup:
25.10.19 15:55:25.273 LISTENER ( WARN ) : initializing module portal_groups
25.10.19 16:37:56.854 LISTENER ( WARN ) : finished initializing module portal_groups with rv=0
Performance of json implementations:
Maybe msgpack is a faster solution:
Untested patch in git:fbest/49862-remove-portal-group-cache:
Patch removes the listener and replaces the logic to use raw ldap calls using the memberOf attribute of groups.
Patch could be fine tuned to even make it faster by iterating only over the groups.
TODO: make the ldap dn comparision case insensitive.
While joining a DC Backup to a domain with 52000 users and 11400 groups it took more than an hour to initialize the portal_groups module
21.08.20 15:32:36.250 LISTENER ( WARN ) : initializing module portal_groups
21.08.20 16:37:21.806 LISTENER ( WARN ) : finished initializing module portal_groups with rv=0