Bug 51236 – replication of large groups (>100000 members) is very slow

Bug 51236 - replication of large groups (>100000 members) is very slow


Summary:	replication of large groups (>100000 members) is very slow

Status:	CLOSED FIXED

Product:	UCS
Classification:	Unclassified
Component:	LDAP
Version:	UCS 4.4
Hardware:	Other Linux

Importance:	P5 critical (vote)
Target Milestone:	UCS 4.4-4-errata
Assigned To:	Julia Bremer
QA Contact:	Erik Damrose

URL:
Keywords:

Depends on:
Blocks:
	Show dependency tree / graph

Reported:	2020-05-06 20:53 CEST by Ralf Haferkamp
Modified:	2020-05-27 12:12 CEST (History)
CC List:	7 users (show)

See Also:	51233
What kind of report is it?:	Bug Report
What type of bug is this?:	5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?:	3: Will affect average number of installed domains
How will those affected feel about the bug?:	2: A Pain – users won’t like this once they notice it
User Pain:	0.171
Enterprise Customer affected?:	Yes
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:

Attachments
suggested patch (3.24 KB, patch) 2020-05-06 20:53 CEST, Ralf Haferkamp	Details \| Diff
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Ralf Haferkamp 2020-05-06 20:53:36 CEST

Created attachment 10350 [details]
suggested patch

We're seeing an issue with the replication of the domain users group. It is really slow and takes a couple of minutes to get a single change replicated for that group (which is currently a bit more than 100000).

I added some debug logging to replication.py and found that it spends a huge amount of time in a tight loop comparing to dicts. Just to print out a log message when the dicts differ. (Which isn't even logged at the default log level.

I'll attach a patch which we have been trying on our test system.

If my analysis is correct an this loop has no other function than to create a log message it might make sense to even move that to the DEBUG level. Or get rid of it completely.

Just for completeness: The fix for bug#48545 is already deployed here.

Comment 1 Daniel Tröder

2020-05-15 12:33:59 CEST

Please see also Bug #51233 (fix already exists in branch) about slow handling of large groups.

Comment 3 Julia Bremer

2020-05-26 11:38:38 CEST

Successful build
Package: univention-directory-replication
Version: 12.0.0-11A~4.4.0.202005201834
Branch: ucs_4.4-0
Scope: errata4.4-4
User: jbremer

d3df32a868 Bug #51236: yaml
02b27123d1 Bug #51236: patch apply


I applied the patch and build univention-directory-replication for 4.4-4 errata

Comment 4 Erik Damrose

2020-05-26 16:31:37 CEST

OK: Patch applied. We did not discuss moving the output to a different loglevel.
OK: yaml
OK: univention-directory-replication 12.0.0-11A~4.4.0.202005201834
Waiting for automated test results before verifying...

Comment 5 Erik Damrose

2020-05-26 17:05:07 CEST

Thanks @Julia, i did not realize there were already test runs since the package was build with the patch.
Verified

Comment 6 Erik Damrose

2020-05-27 12:12:47 CEST

<http://errata.software-univention.de/ucs/4.4/615.html>