Bug 51236

Summary:	replication of large groups (>100000 members) is very slow
Product:	UCS	Reporter:	Ralf Haferkamp <rhafer>
Component:	LDAP	Assignee:	Julia Bremer <bremer>
Status:	CLOSED FIXED	QA Contact:	Erik Damrose <damrose>
Severity:	critical
Priority:	P5	CC:	best, bremer, gulden, heidelberger, michelsmidt, steuwer, troeder
Version:	UCS 4.4
Target Milestone:	UCS 4.4-4-errata
Hardware:	Other
OS:	Linux
See Also:	https://forge.univention.org/bugzilla/show_bug.cgi?id=51233
What kind of report is it?:	Bug Report	What type of bug is this?:	5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?:	3: Will affect average number of installed domains	How will those affected feel about the bug?:	2: A Pain – users won’t like this once they notice it
User Pain:	0.171	Enterprise Customer affected?:	Yes
School Customer affected?:		ISV affected?:
Waiting Support:		Flags outvoted (downgraded) after PO Review:
Ticket number:		Bug group (optional):
Max CVSS v3 score:
Attachments:	suggested patch

Description Ralf Haferkamp 2020-05-06 20:53:36 CEST

Created attachment 10350 [details]
suggested patch

We're seeing an issue with the replication of the domain users group. It is really slow and takes a couple of minutes to get a single change replicated for that group (which is currently a bit more than 100000).

I added some debug logging to replication.py and found that it spends a huge amount of time in a tight loop comparing to dicts. Just to print out a log message when the dicts differ. (Which isn't even logged at the default log level.

I'll attach a patch which we have been trying on our test system.

If my analysis is correct an this loop has no other function than to create a log message it might make sense to even move that to the DEBUG level. Or get rid of it completely.

Just for completeness: The fix for bug#48545 is already deployed here.

Comment 1 Daniel Tröder

2020-05-15 12:33:59 CEST

Please see also Bug #51233 (fix already exists in branch) about slow handling of large groups.

Comment 3 Julia Bremer

2020-05-26 11:38:38 CEST

Successful build
Package: univention-directory-replication
Version: 12.0.0-11A~4.4.0.202005201834
Branch: ucs_4.4-0
Scope: errata4.4-4
User: jbremer

d3df32a868 Bug #51236: yaml
02b27123d1 Bug #51236: patch apply


I applied the patch and build univention-directory-replication for 4.4-4 errata

Comment 4 Erik Damrose

2020-05-26 16:31:37 CEST

OK: Patch applied. We did not discuss moving the output to a different loglevel.
OK: yaml
OK: univention-directory-replication 12.0.0-11A~4.4.0.202005201834
Waiting for automated test results before verifying...

Comment 5 Erik Damrose

2020-05-26 17:05:07 CEST

Thanks @Julia, i did not realize there were already test runs since the package was build with the patch.
Verified

Comment 6 Erik Damrose

2020-05-27 12:12:47 CEST

<http://errata.software-univention.de/ucs/4.4/615.html>