Bug 22852 - Cyrus does not expire deliver-db anymore
Cyrus does not expire deliver-db anymore
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Mail
UCS 4.2
Other Linux
: P2 normal (vote)
: UCS 4.2-0-errata
Assigned To: Daniel Tröder
Sönke Schwardt-Krummrich
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-06-27 11:04 CEST by Sönke Schwardt-Krummrich
Modified: 2017-05-10 15:17 CEST (History)
2 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 4: Minor Usability: Impairs usability in secondary scenarios
Who will be affected by this bug?: 1: Will affect a very few installed domains
How will those affected feel about the bug?: 2: A Pain – users won’t like this once they notice it
User Pain: 0.046
Enterprise Customer affected?: Yes
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sönke Schwardt-Krummrich univentionstaff 2011-06-27 11:04:34 CEST
Quelle: https://bugs.open-xchange.com/show_bug.cgi?id=19603

In the templatefile /etc/univention/templates/files/etc/imapd/cyrus.conf 
there are 2 calls to "/usr/sbin/ctl_deliver" with the Option "-E 3". All
calls to this programs aren't executed successfully. 
"/usr/sbin/ctl_deliver" claims to be depricated for the option "-E" and 
wants to execute "/usr/lib/cyrus/bin/cyr_expire", which does not live in
the expected directory.

-----------------------------------------------------------------------
su - cyrus -c "/usr/sbin/ctl_deliver -E 3"
ctl_deliver -E is deprecated: using cyr_expire -E instead
-----------------------------------------------------------------------

-----------------------------------------------------------------------
...
geteuid()                               = 111
write(2, "ctl_deliver -E is deprecated: usi"..., 58) = 58
execve("/usr/lib/cyrus/bin/cyr_expire", ["/usr/lib/cyrus/bin/cyr_expire", 
   "-E", "3"], [/* 19 vars */]) = -1 ENOENT (No such file or directory)
exit_group(29)                          = ?
-----------------------------------------------------------------------
(made a call to univention with 
Ticket#2011062310001709] "cyr_deliver -E 3 --> cyr_expire -E 3 schlägt fehl: Kein Aufräumen". 

Furthermore in the template, not all cyrus-tools are equipped with the -C Option, 
which have to be given if you use an alternative configurationfile as you 
(or univention) do. So as an attachment you will find the templatefile
"cyrus.conf-corrected". 
In this file I replaced the calls to "ctl_deliver" with "cyr_expire", added the 
"-C"-option to _any_ call in the template, where the command needs this option and  
consequently set this option as the first option. Below the diff:

6c6
<       recover         cmd="/usr/sbin/ctl_cyrusdb -r"
---
>       recover         cmd="/usr/sbin/ctl_cyrusdb -C /etc/imapd/imapd.conf -r"
9c9
<       idled           cmd="idled"
---
>       idled           cmd="idled -C /etc/imapd/imapd.conf"
14c14
<       #mupdatepush   cmd="/usr/sbin/ctl_mboxlist -m"
---
>       #mupdatepush   cmd="/usr/sbin/ctl_mboxlist -C /etc/imapd/imapd.conf -m"
17c17
<       delprune        cmd="/usr/sbin/ctl_deliver -E 3"
---
>       delprune        cmd="/usr/sbin/cyr_expire -C /etc/imapd/imapd.conf -E 3"
19c19
<       tlsprune        cmd="/usr/sbin/tls_prune"
---
>       tlsprune        cmd="/usr/sbin/tls_prune -C /etc/imapd/imapd.conf"
94c94
<       checkpoint      cmd="/usr/sbin/ctl_cyrusdb -c -C /etc/imapd/imapd.conf" period=30
---
>       checkpoint      cmd="/usr/sbin/ctl_cyrusdb -C /etc/imapd/imapd.conf -c" period=30
97c97
<       delprune        cmd="/usr/sbin/ctl_deliver -E 3 -C /etc/imapd/imapd.conf" at=0401
---
>       delprune        cmd="/usr/sbin/cyr_expire -C /etc/imapd/imapd.conf -E 3" at=0401


Steps to Reproduce:
1. Just check the mail.log of an OXforUCS-system for calls to cyr_expire.

Actual Results:
no calls to cyr_expire

Expected Results:
at least calls to cyr_expire at 0401 as shown in the protocol 
below (corrected cyrus.conf applied). Additionally calls to cyr_expire
with the start of cyrus.

-------------------------------------------------------------------------
Jun 25 04:01:00 ox4ucs23 master[4569]: about to exec /usr/sbin/tls_prune
Jun 25 04:01:00 ox4ucs23 master[4570]: about to exec /usr/sbin/cyr_expire
Jun 25 04:01:00 ox4ucs23 cyrus/tls_prune[4569]: tls_prune: purged 0 out of 89 entries
Jun 25 04:01:00 ox4ucs23 master[9779]: process 4569 exited, status 0
Jun 25 04:01:00 ox4ucs23 cyrus/cyr_expire[4570]: duplicate_prune: pruning back 3 days
Jun 25 04:01:00 ox4ucs23 cyrus/cyr_expire[4570]: duplicate_prune: purged 0 out of 26 entries
Jun 25 04:01:00 ox4ucs23 cyrus/cyr_expire[4570]: expunged 0 out of 0 messages from 0 mailboxes
Jun 25 04:01:00 ox4ucs23 master[9779]: process 4570 exited, status 0
-------------------------------------------------------------------------
Comment 1 Sönke Schwardt-Krummrich univentionstaff 2012-01-04 11:58:56 CET
Mit UCS 3.0 weiterhin gültig. Die Deliver-DB wird nicht expired.
Comment 2 Janis Meybohm univentionstaff 2012-09-11 10:30:32 CEST
Erneut aufgefallen an 2012091121000928

Btw.: Bei uns ist die deliver.db fast 6GB groß...
Comment 3 Janis Meybohm univentionstaff 2015-01-13 10:05:07 CET
Ticket#2014120421000459

Looks like as if the missing cyr_expire call leads to problems while migration to cyrus 2.4. On the first start of cyrus 2.4, cyr_expire is called and the startup waits for the process to finish.
That may take several hours depending on the spool size (took like an hour on a slower system with 2.5GB mail spool).
Comment 4 Sönke Schwardt-Krummrich univentionstaff 2015-07-24 12:11:16 CEST
(In reply to Janis Meybohm from comment #3)
> Looks like as if the missing cyr_expire call leads to problems while
> migration to cyrus 2.4. On the first start of cyrus 2.4, cyr_expire is
> called and the startup waits for the process to finish.
> That may take several hours depending on the spool size (took like an hour
> on a slower system with 2.5GB mail spool).

Happened again at a customer with larger mailspool.
/etc/imapd/cyrus.conf contains 2 entries for "delprune":
one in the EVENTS section that calls ctl_deliver resp. cyr_expire each day at 04:01 and one entry in the START section, that does call this command during startup of cyrus.

It should be sufficient to remove/disable the entry in the START section.
Comment 5 Daniel Tröder univentionstaff 2017-05-08 13:32:13 CEST
Solved as part for Bug #44424: in r79177.
Comment 6 Sönke Schwardt-Krummrich univentionstaff 2017-05-08 13:50:22 CEST
We should keep this bug as separate entry.
Added a new entry to the advisory files univention-mail-cyrus.yaml.

univention-mail-cyrus.yaml:
r79183 | Bug #22852: added advisory
Comment 7 Sönke Schwardt-Krummrich univentionstaff 2017-05-08 14:00:01 CEST
The delprune entry should be configured via UCR (enable/disable). By default the delprune entry should be disabled. 

Starting with UCS 4.2-1 we will enable delprune.

Activating the delprune entry may cause IMAP downtimes of several hours depending on the DB size.
Comment 8 Daniel Tröder univentionstaff 2017-05-09 11:15:19 CEST
r79223: fix path for cyr_expire and add UCRVs mail/cyrus/duplicate-supression/expiry/*
r79229: advisory update

UCRVs mail/cyrus/duplicate-supression/expiry/start (default=no) and mail/cyrus/duplicate-supression/expiry/event (default=no) were added.
Comment 9 Sönke Schwardt-Krummrich univentionstaff 2017-05-09 22:26:49 CEST
univention-mail-cyrus (9.0.0-8):
r79258 | Bug #22852: updated UCR variable descriptions

univention-mail-cyrus.yaml:
r79257 | Bug #22852: small adjustments to advisory


OK: code change (updated UCR variable description)
OK: advisory
OK: functional test

root@slave32:/etc# ucr set mail/cyrus/duplicate-suppression/expiry/start=yes
Setting mail/cyrus/duplicate-suppression/expiry/start
Module: create-archivefolder
Multifile: /etc/postfix/ldap.sharedfolderlocal
Multifile: /etc/imapd/cyrus.conf

root@slave32:/etc# systemctl status cyrus-imapd.service
● cyrus-imapd.service - Cyrus IMAP/POP3 daemons
   Loaded: loaded (/lib/systemd/system/cyrus-imapd.service; disabled)
   Active: active (running) since Fr 2017-03-31 07:55:59 CEST; 2s ago
  Process: 27943 ExecStartPre=/usr/sbin/cyrus init-helper start (code=exited, status=0/SUCCESS)
 Main PID: 27959 (cyrmaster)
   CGroup: /system.slice/cyrus-imapd.service
           ├─27959 /usr/sbin/cyrmaster -l 32 -C /etc/imapd/imapd.conf -M /etc/imapd/cyrus.conf
           └─27963 idled -C /etc/imapd/imapd.conf

Mär 31 07:55:59 slave32 cyrus/master[27962]: about to exec /usr/lib/cyrus/bin/idled
Mär 31 07:55:59 slave32 cyrus/master[27964]: about to exec /usr/lib/cyrus/bin/cyr_expire
Mär 31 07:55:59 slave32 cyrus/cyr_expire[27964]: skiplist: checkpointed /var/lib/cyrus/deliver.db (6 records, 972 bytes) in 0 seconds
Mär 31 07:55:59 slave32 cyrus/cyr_expire[27964]: Expunged 0 out of 3 messages from 42 mailboxes
Mär 31 07:55:59 slave32 cyrus/cyr_expire[27964]: duplicate_prune: pruning back 3.00 days
Mär 31 07:55:59 slave32 cyrus/master[27965]: about to exec /usr/lib/cyrus/bin/tls_prune
Mär 31 07:55:59 slave32 cyrus/master[27959]: ready for work
Mär 31 07:55:59 slave32 cyrus/master[27966]: about to exec /usr/lib/cyrus/bin/ctl_cyrusdb
Mär 31 07:55:59 slave32 cyrus/ctl_cyrusdb[27966]: checkpointing cyrus databases
Mär 31 07:55:59 slave32 cyrus/master[27959]: process 27966 exited, status 0
root@slave32:/etc#
Comment 10 Janek Walkenhorst univentionstaff 2017-05-10 15:17:03 CEST
<http://errata.software-univention.de/ucs/4.2/12.html>