Univention Bugzilla – Full Text Bug Listing |
Summary: | replication.py: very slow for groups with lots of members (domain users) | ||
---|---|---|---|
Product: | UCS | Reporter: | Felix Botner <botner> |
Component: | LDAP | Assignee: | Philipp Hahn <hahn> |
Status: | CLOSED FIXED | QA Contact: | Arvid Requate <requate> |
Severity: | normal | ||
Priority: | P5 | CC: | best, hahn, scheinig, schwardt, steuwer, voelker |
Version: | UCS 4.4 | ||
Target Milestone: | UCS 4.4-4-errata | ||
Hardware: | Other | ||
OS: | Linux | ||
See Also: |
https://forge.univention.org/bugzilla/show_bug.cgi?id=50630 https://forge.univention.org/bugzilla/show_bug.cgi?id=50629 https://forge.univention.org/bugzilla/show_bug.cgi?id=45310 https://forge.univention.org/bugzilla/show_bug.cgi?id=46590 https://forge.univention.org/bugzilla/show_bug.cgi?id=52175 |
||
What kind of report is it?: | Bug Report | What type of bug is this?: | 5: Major Usability: Impairs usability in key scenarios |
Who will be affected by this bug?: | 3: Will affect average number of installed domains | How will those affected feel about the bug?: | 5: Blocking further progress on the daily work |
User Pain: | 0.429 | Enterprise Customer affected?: | Yes |
School Customer affected?: | ISV affected?: | ||
Waiting Support: | Yes | Flags outvoted (downgraded) after PO Review: | |
Ticket number: | 2020032321000375 | Bug group (optional): | |
Max CVSS v3 score: | |||
Bug Depends on: | 51061, 51093 | ||
Bug Blocks: |
Description
Felix Botner
2019-01-30 10:16:06 CET
Happens in a customer environment with hourly updates of groups. This is caused by - replication.py doing a (REPLACE, "uniqueMember", […])- - this triggers the LDAP overlay module "memberOf" - it iterates over all those users - and does a dummy update of each user - slapd with LMDB does a fdatasync() for each such update - this is very slow on systems not having a fast persistent storage In some cases this takes longer then 5 minutes, which leads to the LDAP connection to the master (or local slapd?) being closed meanwhile. The next transaction then aborts with TIMEOUT(). This was hidden by Bug #51061. openldap/servers/slapd/overlays/memberof.c works faster if it gets an incremental update using ADD/DELETE instead of the REPLACE. A similar bug has been fixed in ADC Bug #50630 and is pending for S4C Bug #50629 [4.4-4] 5d0a78710c Bug #48545 repl: Do incremental updates for group.uniqueMember .../debian/changelog | 6 ++++++ .../replication.py | 24 ++++++++++++++++++++-- 2 files changed, 28 insertions(+), 2 deletions(-) Package: univention-directory-replication Version: 12.0.0-7A~4.4.0.202004151819 Branch: ucs_4.4-0 Scope: errata4.4-4 [4.4-4] 9f29f95f2b Bug #51093: univention-directory-replication 12.0.0-7A~4.4.0.202004151819 doc/errata/staging/univention-directory-replication.yaml | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) QA: replication.py:modlist() augmented with @ud.trace(with_return=True,repr=str) … DEBUG_END : /usr/lib/univention-directory-listener/system/replication.py.modlist(...): […(0, 'uniqueMember', ['uid=nscd07cf,cn=users,dc=phahn,dc=dev'])] … DEBUG_END : /usr/lib/univention-directory-listener/system/replication.py.modlist(...): […(1, 'uniqueMember', ['uid=nscd07cf,cn=users,dc=phahn,dc=dev'])] Verified: * Code change works * Strictly speaking 'uniqueMember' is just the default for the 'memberof-member-ad' option, which can be overridden via UCR ldap/overlay/memberof/memberof, see /etc/univention/templates/files/etc/ldap/slapd.conf.d/41univention-ldap-overlay-memberof , but your current change is ok for me now. * Advisory Ok PS1: we decided to compare the DNs case-sensitive: even if there only is a change in case, a DELETE/ADD is performed nevertheless. This is desired as we want the Backup/Slave LDAPs to have the same content as the Master. See Bug #46590 where this is already a problem. PS2: As already documented in <https://help.univention.com/t/memberof-attribute-group-memberships-of-user-and-computer-objects/6439> "/usr/share/univention-ldap-overlay-memberof/univention-update-memberof" must be called on all roles after the overlay module "memberof" has been enabled: The new behavior now filters out the null-change and no longer triggers running the overlay module on backups/slaves when the script runs on the master. The script already works on Backups/Slaves as it uses uldap.getRootDnConnection(), which either uses "cn=admin" or "cn=update" depending on the server role. [4.4-4] 929762b894 Bug #48545: Fix incremental updates for group.uniqueMember management/univention-directory-replication/debian/changelog | 6 ++++++ management/univention-directory-replication/replication.py | 6 +++--- 2 files changed, 9 insertions(+), 3 deletions(-) [4.4-4] e8e1580828 Bug #48545: Fix incremental updates for group.uniqueMember .../univention-directory-replication/debian/changelog | 6 ++++++ management/univention-directory-replication/replication.py | 13 ++++++------- 2 files changed, 12 insertions(+), 7 deletions(-) Package: univention-directory-replication Version: 12.0.0-9A~4.4.0.202004161212 Branch: ucs_4.4-0 Scope: errata4.4-4 [4.4-4] 0dde5024c7 Bug #51093: univention-directory-replication 12.0.0-9A~4.4.0.202004161212 doc/errata/staging/univention-directory-replication.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) Verified: * Jenkins tests of last night Ok on Backup & Slave * Artificial case changes in group.uniqueMember are replicated correctly * The UCR variable ldap/overlay/memberof/member is considered correctly (tested by setting it to foo and restarting the listener) * Advisory |