Bug 48729 - Add tool to clean up old cn=translog entries
Add tool to clean up old cn=translog entries
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Notifier (univention-directory-notifier)
UCS 4.3
Other Linux
: P5 major (vote)
: UCS 4.4-0-errata
Assigned To: Philipp Hahn
Arvid Requate
:
Depends on: 48427
Blocks: 49430 49505 49872
  Show dependency treegraph
 
Reported: 2019-02-21 19:28 CET by Arvid Requate
Modified: 2019-08-14 14:54 CEST (History)
3 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 4: Will affect most installed domains
How will those affected feel about the bug?: 5: Blocking further progress on the daily work
User Pain: 0.571
Enterprise Customer affected?: Yes
School Customer affected?:
ISV affected?:
Waiting Support: Yes
Flags outvoted (downgraded) after PO Review:
Ticket number: 2019042221000396
Bug group (optional):
Max CVSS v3 score: 5.3 (CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N/E:H)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Arvid Requate univentionstaff 2019-02-21 19:28:28 CET
We should have a tool to remove entries from cn=translog with an id that is lower than a specified value (or maybe something like "keep only the last X entries").
Comment 1 Christian Völker univentionstaff 2019-04-24 12:34:53 CEST
Currently happening on customer site.

Database has been set recently to 8GB approx. two weeks ago and has reached again the size limit.
Customer increased again (now to ~17G). The issue is not about empty pages, the translog already uses 48.6% of the 17GB:

=======================================================================
root@ucs-master:~# used_pages=$(mdb_stat -e /var/lib/univention-ldap/translog | sed -n 's/^ *Number of pages used: //p')
root@ucs-master:~# max_pages=$(mdb_stat -e /var/lib/univention-ldap/translog | sed -n 's/^ *Max pages: //p')
root@ucs-master:~# python -c "print('%.1f used' % (float($used_pages) / $max_pages * 100))"
48.6 used

root@ucs-master:~# ucr get ldap/database/mdb/maxsize
17179869184
=======================================================================

An unlimited growth of the database will cause severe issues at some stage (/-fs usage and so on).


As no workaround available and customer disk is getting filled up tagging as "waiting support"
Comment 2 Philipp Hahn univentionstaff 2019-05-03 13:25:25 CEST
[4.4-0] 66b3ff5ac8 Bug #48729 udn: Implement prune command
[4.4-0] a6dbca4664 Bug #48729 udn: Implement prune command
 .../univention-directory-notifier/debian/changelog |   6 ++
 .../univention-translog                            | 119 ++++++++++++++++++++-
 2 files changed, 124 insertions(+), 1 deletion(-)
[4.4-0] 8d46c51abb Bug #48729 udn: Add documentation
 .../univention-translog                            | 123 +++++++++++++++++++--
 1 file changed, 114 insertions(+), 9 deletions(-)
[4.4-0] 756502a095 Bug #48729 udn: Reuse index.remove() code
 .../univention-translog                            | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
[4.4-0] 2411f0f436 Bug #48729 udn: Reuse restart() code
 management/univention-directory-notifier/univention-translog | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
[4.4-0] eaafcdbd35 Bug #48729 udn: Fix indention
 .../univention-translog                            | 172 ++++++++++-----------
 1 file changed, 86 insertions(+), 86 deletions(-)

Package: univention-directory-notifier
Version: 13.0.1-15A~4.4.0.201905031308
Branch: ucs_4.4-0
Scope: errata4.4-0

[4.4-0] cfa10faa12 Bug #48729: univention-directory-notifier 13.0.1-15A~4.4.0.201905031308
 doc/errata/staging/univention-directory-notifier.yaml | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)



Prune every transaction before 10:
 ./univention-translog -vvvv prune 10
Keep only the last 10 transactions:
 ./univention-translog -vvvv prune -10
Comment 3 Arvid Requate univentionstaff 2019-05-07 19:45:32 CEST
Ok works, two minor points:

* Is this a typo in the log message or what does that mean:
  log.info('Keeeling last %d transactions before %d', -opt.trans, translog.last)

* I have to specify at least two -v to see any info messages. I would expect
  to see some output when I specify --verbose once:

root@master10:~# /usr/share/univention-directory-notifier/univention-translog -v --dry-run prune 102
root@master10:~# /usr/share/univention-directory-notifier/univention-translog -vv --dry-run prune 102
2019-05-02 14:53:23,648:INFO:Index of size 13265 contains 1473 entries
2019-05-02 14:53:23,648:INFO:Purging/keeping 100..102..1472
2019-05-02 14:53:23,684:INFO:Deleted reqSession=100,cn=translog
2019-05-02 14:53:23,685:INFO:Deleted reqSession=101,cn=translog

* Idea for improvement for later: Backup removed translog file entries in a compressed file. But I would not do this now.

* Advisory: Ok

Reopening for you comments or changes.
Comment 4 Philipp Hahn univentionstaff 2019-05-08 08:27:08 CEST
(In reply to Arvid Requate from comment #3)
> * Is this a typo in the log message or what does that mean:
>   log.info('Keeeling last %d transactions before %d', -opt.trans,
> translog.last)

Fixed

> * I have to specify at least two -v to see any info messages. I would expect
>   to see some output when I specify --verbose once:

Increase default debug level.

> * Idea for improvement for later: Backup removed translog file entries in a
> compressed file. But I would not do this now.

The old file is already moved to .bak

[4.4-0] 0ce117fe13 Bug #48729 udl: Fix spelling errors and increase verbosity
 management/univention-directory-notifier/debian/changelog    | 6 ++++++
 management/univention-directory-notifier/univention-translog | 6 +++---
 2 files changed, 9 insertions(+), 3 deletions(-)

Package: univention-directory-notifier
Version: 13.0.1-16A~4.4.0.201905080824
Branch: ucs_4.4-0
Scope: errata4.4-0

[4.4-0] 1571569321 Bug #48729: univention-directory-notifier 13.0.1-16A~4.4.0.201905080824
 doc/errata/staging/univention-directory-notifier.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 5 Arvid Requate univentionstaff 2019-05-08 12:38:56 CEST
Verified:
* Prune transactions older than 10: /usr/share/univention-directory-notifier/univention-translog -v --dry-run prune 10
* Prune transactions, keeping the last 1000 : /usr/share/univention-directory-notifier/univention-translog -v --dry-run prune -1000 ## negative number

Other small improvements for the log messages are postponed to Bug #49430
Comment 6 Arvid Requate univentionstaff 2019-05-08 13:26:18 CEST
<http://errata.software-univention.de/ucs/4.4/81.html>
Comment 7 Christian Völker univentionstaff 2019-05-20 09:10:56 CEST
Attempt to use the new prune command on Master UCS to reduce the size of the translog database. The translog database has reached almost 87% capacity (of 16GB). 


Initially attempt to keep only the last 1,000,000 entries (of 15,538,649) using the following command:
root@ucs-master:~# /usr/share/univention-directory-notifier/univention-translog -v prune -1000000

Right at Deleted reqSession=399999,cn=translog the following error was thrown:

Traceback (most recent call last):
  File "/usr/share/univention-directory-notifier/univention-translog", line 1406, in <module>
    exit(main())
  File "/usr/share/univention-directory-notifier/univention-translog", line 420, in main
    return opt.func(opt) or 0
  File "/usr/share/univention-directory-notifier/univention-translog", line 1182, in prune
    prune_ldap(opt)
  File "/usr/share/univention-directory-notifier/univention-translog", line 1237, in prune_ldap
    rtype, rdata, rmsgid, serverctrls = ld.result3(response)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 514, in result3
    resp_ctrl_classes=resp_ctrl_classes
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 521, in result4
    ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 106, in _ldap_call
    result = func(*args,**kwargs)
ldap.SIZELIMIT_EXCEEDED: {'desc': 'Size limit exceeded'}


Re-running the exact same prune command: CRITICAL:Already purged.


Restoring the snapshot.

Attempt pruning a smaller amount of entries at one time (keeping 15,500,000 of 15,538,649). This attempt also resulted in the same timeout error as above.