Bug 43937 - systemd fails to start system with separate /usr/ file system
systemd fails to start system with separate /usr/ file system
Product: UCS
Classification: Unclassified
Component: General
UCS 4.2
Other Linux
: P5 critical (vote)
: UCS 4.2
Assigned To: Philipp Hahn
Stefan Gohmann
: interim-3, systemd
Depends on:
Blocks: 43330 44347
  Show dependency treegraph
Reported: 2017-03-17 16:48 CET by Philipp Hahn
Modified: 2017-09-26 10:47 CEST (History)
3 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hahn univentionstaff 2017-03-17 16:48:30 CET
systemd does not start a system with a separate /usr/ - see URL.
This leads to all kinds of services failing.
Using "systemd.debug-shell=1" to get a debug shell and then
>$ systemctl list-jobs
>JOB UNIT                                             TYPE    STATE
>  1 graphical.target                                 start   waiting
>  2 multi-user.target                                start   waiting
> 77 systemd-update-utmp-runlevel.service             start   waiting
> 83 apache2.service                                  start   waiting
> 86 postfix.service                                  start   waiting
> 87 mail-transport-agent.target                      start   waiting
> 88 univention-directory-listener.service            start   waiting
> 92 univention-maintenance.service                   start   waiting
> 93 univention-management-console-web-server.service start   waiting
basically network failed to start as they are on /usr/; as UCR is on /usr/ as well, this breaks lots of other services.
>105 plymouth-quit-wait.service                       start   waiting
>108 plymouth-quit.service                            start   waiting
>110 getty.target                                     start   waiting
>111 getty@tty1.service                               start   waiting
so you don't even get a getty - waits for plymouth-quit-wait.service 
>201 dbus.service                                     start   waiting
>283 systemd-logind.service                           restart waiting
>366 ssh.service                                      restart waiting
>448 rsyslog.service                                  restart waiting

To me it looks like "systemd" expects the InitRamFS to already mount /usr/, but Debian's initrams doesn't.
OK: /run/systemd/generator.usr.mount is there
??: /run/systemd/generator/initrd-root-fs.target.requires/usr.mount is there, but seems to go unused.
But this happens only *after* the initramfs is done - inside the initramfs systemd is *not* used.
systemd in initramfs is not supported by Debian: <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717429>

See <https://www.freedesktop.org/software/systemd/man/bootup.html#Bootup%20in%20the%20Initial%20RAM%20Disk%20(initrd)>.
There was a discussion at DebConf15 to replace initramfs-tools from Debian with "dracut", as the former is only used by Debian and is not actively maintained any more.
Comment 1 Philipp Hahn univentionstaff 2017-03-20 16:04:11 CET

r77986 | Bug #43937 up: Block upgrade with separate /usr
r77985 | Bug #43937 up: Fix whitespace

Package: univention-updater
Version: 12.0.4-7A~
Branch: ucs_4.2-0

QA: Either install UCS-4.1 with a separate /usr or do some bind-mount magic:
telinit S
swapoff -a
sed -i '/swap/d' /etc/fstab
mke2fs /dev/mapper/vg_ucs-swap_1
DIRS=$(cd /usr && ls -d */)
echo "/usr /mnt none bind 0 0" >>/etc/fstab
echo "/dev/mapper/vg_ucs-swap_1 /usr ext4 ro 0 0" >>/etc/fstab
for d in $DIRDS;do echo "/mnt/$d /usr/$d none bind 0 0";done >>/etc/fstab
mount /mnt
mount -o rw /usr
for d in $DIRS;do mkdir /usr/$d;done
mount -o remount,ro /usr
for d in $DIRS;do mount /usr/$d;done
telinit 2
Comment 2 Stefan Gohmann univentionstaff 2017-03-21 16:43:26 CET
As discussed, please use "init 1" instead of "telinit S". Everything else works.
Comment 3 Philipp Hahn univentionstaff 2017-03-21 16:48:28 CET
(In reply to Stefan Gohmann from comment #2)
> As discussed, please use "init 1" instead of "telinit S". Everything else
> works.

Already fixed.
Comment 4 Stefan Gohmann univentionstaff 2017-03-21 18:48:56 CET
Comment 5 Erik Damrose univentionstaff 2017-03-30 09:56:04 CEST
Reopen: The link in preup to help.univention.com is no longer valid, probably due to the maintenance the website had. In a quick search the original thread no longer exists
Comment 6 Philipp Hahn univentionstaff 2017-03-30 11:39:19 CEST
r78490 | Bug #43937 up: Block upgrade with separate /usr

Package: univention-updater
Version: 12.0.5-3A~
Branch: ucs_4.2-0
Comment 7 Stefan Gohmann univentionstaff 2017-03-31 14:20:45 CEST
Comment 8 Stefan Gohmann univentionstaff 2017-04-04 18:29:55 CEST
UCS 4.2 has been released:

If this error occurs again, please use "Clone This Bug".