Univention Bugzilla – Bug 49505
univention-translog purge: Size limit exceeded
Last modified: 2019-06-12 16:22:56 CEST
+++ This bug was initially created as a clone of Bug #48729 +++ Attempt to use the new prune command on Master UCS to reduce the size of the translog database. The translog database has reached almost 87% capacity (of 16GB). Initially attempt to keep only the last 1,000,000 entries (of 15,538,649) using the following command: root@ucs-master:~# /usr/share/univention-directory-notifier/univention-translog -v prune -1000000 Right at Deleted reqSession=399999,cn=translog the following error was thrown: Traceback (most recent call last): File "/usr/share/univention-directory-notifier/univention-translog", line 1406, in <module> exit(main()) File "/usr/share/univention-directory-notifier/univention-translog", line 420, in main return opt.func(opt) or 0 File "/usr/share/univention-directory-notifier/univention-translog", line 1182, in prune prune_ldap(opt) File "/usr/share/univention-directory-notifier/univention-translog", line 1237, in prune_ldap rtype, rdata, rmsgid, serverctrls = ld.result3(response) File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 514, in result3 resp_ctrl_classes=resp_ctrl_classes File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 521, in result4 ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop) File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 106, in _ldap_call result = func(*args,**kwargs) ldap.SIZELIMIT_EXCEEDED: {'desc': 'Size limit exceeded'} Re-running the exact same prune command: CRITICAL:Already purged. Restoring the snapshot. Attempt pruning a smaller amount of entries at one time (keeping 15,500,000 of 15,538,649). This attempt also resulted in the same timeout error as above.
Maybe we can/should use mdb_copy -c to defragment: /etc/init.d/univention-directory-notifier stop && install -d -o openldap -g openldap /var/lib/univention-ldap/translog.new && mdb_copy -c /var/lib/univention-ldap/translog /var/lib/univention-ldap/translog.new && rm -rf /var/lib/univention-ldap/translog.bak && mv /var/lib/univention-ldap/translog /var/lib/univention-ldap/translog.bak && mv /var/lib/univention-ldap/translog.new /var/lib/univention-ldap/translog && /etc/init.d/univention-directory-notifier start
(In reply to Philipp Hahn from comment #0) > Attempt to use the new prune command on Master UCS to reduce the size of the > translog database. The translog database has reached almost 87% capacity (of > 16GB). ... > ldap.SIZELIMIT_EXCEEDED: {'desc': 'Size limit exceeded'} The LDAP server has to return more entries then allowed by UCRV `ldap/sizelimit`, which is 400k by default: # ldapsearch -xLLL -H ldapi:// -b cn=config -s base olcSizeLimit olcSizeLimit: 400.000 This is *not* a file-system limit, so no need to copy the mdb-file. The error than would be `ldap.INTERNAL_ Quoting <https://www.openldap.org/doc/admin24/limits.html#Size%20limits%20and%20Paged%20Results>: > However, the size limit applies to the *total* count of entries returned within the search, and *not* to a single page. We should remove the limit for the ldapi:/// connection by adding the following line to <file:///etc/univention/templates/files/etc/ldap/slapd.conf.d/40univention-ldap-server_database>: print 'limits dn.exact="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" size.prtotal=unlimited size.pr=unlimited' For QA testing: Temporarily reduce the size limit to 10k: ucr set ldap/sizelimit=10000 Create 20k entries: b="$(ucr get ldap/base)";for ((i=1;i<10100;i++));do printf 'dn: ou=%d,%s\nou: %d\nobjectClass: organizationalUnit\n\n' "$i" "$b" "$i";done|ldapadd -QY EXTERNAL -H ldapi:// Verify the size limit: ldapsearch -Y EXTERNAL -Q -H ldapi:/// -b cn=translog -s one 1.1 | wc -l Purge 10k entries: /usr/share/univention-directory-notifier/univention-translog -vv prune -10000 ...
[4.4-0] a32a2219aa Bug #49505: Unlimited page size for UDN translog doc/errata/staging/univention-ldap.yaml | 13 +++++++++++++ .../conffiles/etc/ldap/slapd.conf.d/99translog | 1 + management/univention-ldap/debian/changelog | 6 ++++++ 3 files changed, 20 insertions(+) Package: univention-ldap Version: 15.0.0-20A~4.4.0.201906052030 Branch: ucs_4.4-0 Scope: errata4.4-0 [4.4-0] ede20d8d2c Bug #49505: univention-ldap 15.0.0-20A~4.4.0.201906052030 doc/errata/staging/univention-ldap.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) QA: ucr set ldap/sizelimit=25 apt install univention-ldap-{acl-master,server,config}=15.0.0-20A~4.4.0.201906052030 /etc/init.d/slapd restart univention-ldapsearch -LLLo ldif-wrap=no -b cn=translog 1.1 | grep -c ^dn # Size limit exceeded (4) # only returns 25 entries ldapsearch -LLLo ldif-wrap=no -QY EXTERNAL -H ldapi:/// -b cn=translog -E pr=10/noprompt 1.1 | grep -c ^dn # returns all entries /usr/share/univention-directory-notifier/univention-translog -v prune -10 # should work
Ok, works. bb2b1fc0fc | Fix advisory wording
<http://errata.software-univention.de/ucs/4.4/146.html>