Description Sönke Schwardt-Krummrich 2015-08-27 10:56:11 CEST

Ticket#2015082721000207:
A customer reported error messages from cyrus after updating from UCS 3.2-6 to
UCS 4.0-2:

Aug 27 09:24:01 mail cyrus/cyr_expire[13622]: unable to open mailbox customer.example.com!user.user5
Aug 27 09:24:01 mail cyrus/cyr_expire[13622]: IOERROR: creating directory /run/cyrus/lock/domain/s/customer.example.com/s/user/user5: No space left on device
Aug 27 09:24:01 mail cyrus/cyr_expire[13622]: IOERROR: locking customer.example.com!user.user5.Ham: No space left on device
Aug 27 09:24:01 mail cyrus/cyr_expire[13622]: unable to open mailbox customer.example.com!user.user5.Ham
Aug 27 09:24:01 mail cyrus/cyr_expire[13622]: IOERROR: creating directory /run/cyrus/lock/domain/s/customer.example.com/s/user/user5: No space left on device
Aug 27 09:24:01 mail cyrus/cyr_expire[13622]: IOERROR: locking customer.example.com!user.user5.Spam: No space left on device
Aug 27 09:24:01 mail cyrus/cyr_expire[13622]: unable to open mailbox customer.example.com!user.user5.Spam
Aug 27 09:24:01 mail cyrus/cyr_expire[13622]: IOERROR: locking customer.example.com!user.user6: No space left on device

Each morning around 4 a.m. cyr_expire creates for each of the customers mailboxes and subfolders (around 250.000) a file in /run/cyrus. The size of the filesystem was large enough (10% of RAM == 1.6 GiB) but the number of inodes was nt sufficient (around 189.000 inodes):

root@mail:~# df -h
Dateisystem               Größe Benutzt Verf. Verw% Eingehängt auf
rootfs                     246G    157G   79G   67% /
udev                        10M       0   10M    0% /dev
tmpfs                      1,6G     14M  1,6G    1% /run
/dev/mapper/vg_ucs-rootfs  246G    157G   79G   67% /
tmpfs                      5,0M       0  5,0M    0% /run/lock
tmpfs                      3,2G       0  3,2G    0% /run/shm
/dev/mapper/vg_ucs-log      30G    1,7G   27G    7% /var/log
/dev/sda2                  976M    160M  766M   18% /boot
root@mail:~# df -i
Dateisystem                 Inodes IBenutzt    IFrei IUse% Eingehängt auf
rootfs                    16384000  5711015 10672985   35% /
udev                        182776      352   182424    1% /dev
tmpfs                       189734    46153   453847   10% /run
/dev/mapper/vg_ucs-rootfs 16384000  5711015 10672985   35% /
tmpfs                       189734        6   189728    1% /run/lock
tmpfs                       189734        2   189732    1% /run/shm
/dev/mapper/vg_ucs-log     1966080     1586  1964494    1% /var/log
/dev/sda2                    65536      308    65228    1% /boot
root@mail:~#

root@mail:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=182776,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1662884k,mode=755)
/dev/mapper/vg_ucs-rootfs on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3325760k)
/dev/mapper/vg_ucs-log on /var/log type ext4 (rw,relatime,data=ordered)
/dev/sda2 on /boot type ext3 (rw,relatime,data=ordered)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
root@mail:~#

The number of inodes for /run/ has been increased by adding an entry to /etc/fstab:

tmpfs   /run   tmpfs   rw,nosuid,noexec,relatime,size=10%,nr_inodes=500000,mode=755     0    0

We should check if
- the diagnostics module could check if enough inodes are available on each filesystem
- it is clever to increase the number of inodes for /run/ by default in UCS (increased memory consumption?)
- ...