Bug 41281 - use nosync for mdb during join and slapindex
use nosync for mdb during join and slapindex
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Join (univention-join)
UCS 4.1
Other Linux
: P5 enhancement (vote)
: UCS 4.1-3-errata
Assigned To: Felix Botner
Stefan Gohmann
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2016-05-17 13:38 CEST by Sönke Schwardt-Krummrich
Modified: 2016-09-29 21:21 CEST (History)
3 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 2: Improvement: Would be a product improvement
Who will be affected by this bug?: 3: Will affect average number of installed domains
How will those affected feel about the bug?: 1: Nuisance – not a big deal but noticeable
User Pain: 0.034
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional): UCS Performance
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sönke Schwardt-Krummrich univentionstaff 2016-05-17 13:38:51 CEST
A customer suggested to set the nosync flag for mdb in univention-join while the LDAP is initially replicated. Additionally, nosync should be set while slapindex is called (automatically). Afterwards is should be set to the old state.

The customer reduced the amount of time required for a slapindex from 2 hours down to about 20 seconds.
Comment 1 Florian Best univentionstaff 2016-05-17 19:10:38 CEST
What does the "nosync" flag do?
Comment 2 Markus Dählmann 2016-07-07 09:49:35 CEST
"nosync" turns off syncing the database to disk after every transaction, which is very useful for slapindex, which would otherwise do an fdatasync after indexing every single LDAP object. As Sönke said this reduces the run time of slapindex from 2 hours to a couple of seconds on many of our school slaves.

It would greatly benefit us if /usr/share/univention-ldap/ldap_setup_index would set this parameter before invoking slapindex, especially when we do several index-changing UCS updates in a row.
Comment 3 Markus Dählmann 2016-08-16 16:32:40 CEST
After some research I believe that running "slapindex -q" would basically do the same as setting the "nosync" MDB option. Plus, with '-q' slapindex makes use of the "tool-threads" option, which is already configurable via UCR, so it can run multi-threaded.

Of course, all of this comes at the cost of data integrity in case of a system failure while indexing, but as long as it is not the master LDAP being indexed, this is an acceptable price to pay, since one could just rejoin the failed machine.

So in short: A UCR variable that would let us use "-q" in automated slapindex scenarios would be highly appreciated.
Comment 4 Felix Botner univentionstaff 2016-09-05 18:04:59 CEST
(In reply to Sönke Schwardt-Krummrich from comment #0)
> A customer suggested to set the nosync flag for mdb in univention-join while
> the LDAP is initially replicated. 

works for me with:
-> ucr set ldap/database/mdb/envflags='nosync' && \
   univention-join && \
   ucr unset ldap/database/mdb/envflags && \
   service slapd restart

> slapindex

Added UCR variable ldap/index/quickmode, If true "ldap_setup_index" starts slapindex in "quick" mode.

univention-ldap: 12.1.6-38.835.201609051757

univention-ldap.yaml

merged to 4.2
Comment 5 Stefan Gohmann univentionstaff 2016-09-07 06:38:14 CEST
One small issue: Can you describe in the UCR variable what the default is?

root@master411:~# ucr search ldap/index/quickmode
ldap/index/quickmode: <empty>
 If this option is activated, the LDAP reindex tool slapindex is started in "quick" mode (fewer integrity checks, improved indexing time).

root@master411:~#
Comment 6 Felix Botner univentionstaff 2016-09-07 09:25:20 CEST
OK, univention-ldap-server now sets a default for  ldap/index/quickmode. Added default to description.

errata4.1-3
4.2-0
updated yaml
Comment 7 Stefan Gohmann univentionstaff 2016-09-07 12:16:12 CEST
Code review: OK

ucs-test: OK

Tests: OK

YAML: OK
Comment 8 Janek Walkenhorst univentionstaff 2016-09-07 18:41:40 CEST
<http://errata.software-univention.de/ucs/4.1/249.html>