Bug 52977 - Failed univention-system-setup-boot.service
Failed univention-system-setup-boot.service
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: System setup
UCS 5.0
Other Linux
: P5 normal (vote)
: UCS 5.0
Assigned To: Philipp Hahn
Jürn Brodersen
: interim-2
: 53173 (view as bug list)
Depends on:
Blocks: 53173
  Show dependency treegraph
 
Reported: 2021-03-23 13:57 CET by Philipp Hahn
Modified: 2021-05-25 16:03 CEST (History)
6 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hahn univentionstaff 2021-03-23 13:57:33 CET
After installation from UCS-5.0-0-rc0:

# journalctl  -u  univention-system-setup-boot.service
-- Logs begin at Tue 2021-03-23 11:53:09 CET, end at Tue 2021-03-23 13:55:04 CET. --
Mär 23 11:53:21 unassigned-hostname systemd[1]: Starting LSB: Univention System Setup on boot...
Mär 23 11:53:21 unassigned-hostname univention-system-setup-boot[927]: Starting Univention System Setup on boot.
Mär 23 11:53:21 unassigned-hostname univention-system-setup-boot[927]: error: unexpectedly disconnected from boot status daemon
Mär 23 11:53:22 unassigned-hostname chpasswd[1107]: pam_unix(chpasswd:chauthtok): password changed for __systemsetup__
Mär 23 11:53:33 unassigned-hostname su[1345]: (to nobody) root on none
Mär 23 11:53:33 unassigned-hostname su[1345]: pam_unix(su:session): session opened for user nobody by (uid=0)
Mär 23 11:58:21 unassigned-hostname systemd[1]: univention-system-setup-boot.service: Start operation timed out. Terminating.
Mär 23 11:58:21 unassigned-hostname systemd[1]: univention-system-setup-boot.service: Control process exited, code=killed, status=15/TERM
Mär 23 11:58:21 unassigned-hostname systemd[1]: univention-system-setup-boot.service: Failed with result 'timeout'.
Mär 23 11:58:21 unassigned-hostname systemd[1]: Failed to start LSB: Univention System Setup on boot.
Mär 23 12:30:16 dc0 chpasswd[2055]: pam_unix(chpasswd:chauthtok): password changed for __systemsetup__

Restarting fixes the service, but no USS is shown:

root@dc0:~# systemctl restart univention-system-setup-boot.service
root@dc0:~# systemctl status univention-system-setup-boot.service
● univention-system-setup-boot.service - LSB: Univention System Setup on boot
   Loaded: loaded (/etc/init.d/univention-system-setup-boot; generated)
   Active: active (exited) since Tue 2021-03-23 13:56:44 CET; 8s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 3032 ExecStart=/etc/init.d/univention-system-setup-boot start (code=exited, status=0/SUCCESS)

Mär 23 13:56:43 dc0 systemd[1]: Starting LSB: Univention System Setup on boot...
Mär 23 13:56:43 dc0 univention-system-setup-boot[3032]: Starting Univention System Setup on boot.
Mär 23 13:56:44 dc0 univention-system-setup-boot[3032]: done.
Mär 23 13:56:44 dc0 systemd[1]: Started LSB: Univention System Setup on boot.
Comment 1 Philipp Hahn univentionstaff 2021-03-23 14:43:36 CET
This was *after* the reboot and first USS run, so it looks like univention-system-setup-boot.service is not prepared for taking longer. This is confirmed by the following message:

> systemd[1]: univention-system-setup-boot.service: Start operation timed out. Terminating.

$ systemctl cat univention-system-setup-boot.service →
[Service]
Type=forking
TimeoutSec=5min
ExecStart=/etc/init.d/univention-system-setup-boot start

The init-script calls startwithfirefox in the *forground* (to prevent users to already login to an UCS appliance), but systemd expects the process to return within 5 minutes and to leave behind at least one process.
Comment 2 Ingo Steuwer univentionstaff 2021-03-25 16:50:34 CET
I'm not sure to understand when this happens. Did you configure the system using USS, do a reboot, and afterwards it still tries to start USS?
Comment 4 Philipp Hahn univentionstaff 2021-04-17 17:47:26 CEST
(In reply to Ingo Steuwer from comment #2)
> I'm not sure to understand when this happens. Did you configure the system
> using USS, do a reboot, and afterwards it still tries to start US

1. D-I
2. Reboot
3. USS
4. Login to console
5. univention-system-setup-boot.service is state=failed

generated by systemd-sysv-generator:
 [Service]
 Type=forking
 TimeoutSec=5min

We should switch to systemd completely or as an intermediate ass /lib/systemd/system/univention-system-setup-boot.service.d/bug52977.conf with
 [Service]
 Type=oneshot
 RemainAfterExit=false
 TimeoutSec=infinity

This looks only cosmetic as USS continues running, but it has some bad side effects:
For appliances we do a setup and then abort USS; it downloads the packages from the ISO and puts them into /var/cache/univention-system-setup/packages/.
It then deletes /etc/ssh/ssh_host_*_key* and /etc/univention/ssl/.

When the appliance starts they must be rebuilt before ssh and apache2 can start.
This is expected to be done by /etc/init.d/univention-system-setup-boot-prepare, which no longer works:
systemd reorders the scripts honoring the "Before=" and "After=" constraints to palatalize as much as possible.
Even when "apache2.service" is forced to be after "u-s-s-b-prepare.service" it does not work!
The reason for this is that "cron.service" is started very early and executed the jobs from "/etc/cron.d/univention-updater".
This includes
> @reboot root /usr/share/univention-updater/enable-apache2-umc
which then executes
> service apache2 start
explicitly. This is translated into
> systemctl start apache2.service --ignore-dependencies
which fails as the /etc/univention/ssl/ is still missing.
"apache2.service" then remains in the failed state even when later systemd again tries to start it after the certificate has been generated.

I talked to @fbest and @dwiesent and we decided to remove the logic for (un)masking apache2 by /usr/share/univention-updater/{en,dis}able-apache2-umc.

# ucr get grub/append 
net.ifnames=0 systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M printk.devkmsg=on


[5.0-0] 2b2b1d0b35 fix[uss]: Fix APT repository permissions
 .../usr/share/univention-system-setup/download-packages         | 47 +++++++++++++----------------
 1 file changed, 21 insertions(+), 26 deletions(-)

[5.0-0] 329db0c5b7 fix[updater]: Stop en-/disabling apache2.service
 base/univention-updater/debian/changelog           |  6 ++++
 base/univention-updater/script/disable-apache2-umc | 47 +++++++++++++++++++-----------
 base/univention-updater/script/enable-apache2-umc  | 65 ++++++++++++++++++++++++++++++------------
 3 files changed, 83 insertions(+), 35 deletions(-)

[5.0-0] fedae5c5ba fix[uss]: Convert USS-boot to systemd.service
 base/univention-system-setup/debian/changelog                   |  6 ++
 base/univention-system-setup/debian/compat                      |  1 -
 base/univention-system-setup/debian/control                     |  2 +-
 base/univention-system-setup/debian/rules                       |  9 +--
 .../debian/univention-system-setup-boot.init                    | 91 -----------------------------
 .../debian/univention-system-setup-boot.install                 |  2 +-
 .../debian/univention-system-setup-boot.service                 | 19 ++++++
 ...-setup-boot.univention-system-setup-boot-prepare-ssh.service | 11 ++++
 ...-setup-boot.univention-system-setup-boot-prepare-ssl.service | 11 ++++
 .../share/univention-system-setup-boot/init}                    | 64 +++++++++-----------
 10 files changed, 82 insertions(+), 134 deletions(-)

[5.0-0] 44e1803d3a refacor[uss]: shellcheck
 .../debian/univention-system-setup-boot.init                    | 12 ++--
 .../usr/share/univention-system-setup/startxwithfirefox         | 91 ++++++++++++++++-------------
 2 files changed, 55 insertions(+), 48 deletions(-)

[5.0-0] 837aa246a6 refacor[uss]: restarting services
 base/univention-system-setup/usr/share/univention-system-setup/startxwithfirefox | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)

[5.0-0] c807647151 fix[uss]: Remove EC2 networ handling
 base/univention-system-setup/debian/univention-system-setup-boot.init | 7 -------
 1 file changed, 7 deletions(-)

[5.0-0] 545a77d910 fix[uss]: Remove background setting code
 .../usr/share/univention-system-setup/startxwithfirefox                | 22 +---------------------
 1 file changed, 1 insertion(+), 21 deletions(-)

[5.0-0] 32e6a30a36 fix[uss]: Make boot services exclusive
 .../debian/univention-system-setup-boot.univention-system-setup-boot-prepare-ssl.service     | 1 +
 1 file changed, 1 insertion(+)

Package: univention-updater
Version: 15.0.3-40A~5.0.0.202104161602

Package: univention-system-setup
Version: 13.0.3-21A~5.0.0.202104171727

OK: rm -rf /etc/ssh/ssh_host_* /etc/univention/ssl ; reboot
OK: systemctl --failed
Comment 5 Philipp Hahn univentionstaff 2021-04-20 10:25:21 CEST
[5.0-0] 99707bc2d2 fix[updater]: Stop en-/disabling apache2.service 2
 base/univention-updater/script/enable-apache2-umc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Package: univention-updater
Version: 15.0.3-40A~5.0.0.202104201002

FYI: Bootup is delayed until univention-system-setup-boot-prepase-ssl has re-created the deleted SSL certificaes in Appliance mode. This also seems to affect the installer test → needs more investigation
Comment 6 Philipp Hahn univentionstaff 2021-04-20 17:53:28 CEST
[5.0-0] d079085a7c fix[uss]: Prevent USS from starting
 base/univention-system-setup/debian/rules                                                 | 4 +---
 base/univention-system-setup/debian/univention-system-setup-boot.install                  | 1 +
 .../univention-system-setup-boot-prepare-ssh.service}                                     | 0
 .../univention-system-setup-boot-prepare-ssl.service}                                     | 0
 base/univention-system-setup/{debian => systemd}/univention-system-setup-boot.service     | 0
 5 files changed, 2 insertions(+), 3 deletions(-)

Package: univention-system-setup
Version: 13.0.3-21A~5.0.0.202104201645

OK: Upgrade of uss-boot no longer restart a running USS.
OK: scenarios/autotest-090-master-no-samba.cfg installs
Comment 7 Philipp Hahn univentionstaff 2021-05-06 14:57:20 CEST
*** Bug 53173 has been marked as a duplicate of this bug. ***
Comment 8 Jürn Brodersen univentionstaff 2021-05-12 11:06:57 CEST
What I tested:
Changes -> OK
Installation CD -> OK
Installation appliance -> OK
Reboot during system-setup -> OK
Status "univention-system-setup-boot.service" -> OK
restart "univention-system-setup-boot.service" -> OK
ssh key recreation -> OK
tls key recreation -> OK
jenkins -> OK

No functional changes -> no changelog necessary -> OK
Comment 9 Florian Best univentionstaff 2021-05-25 16:03:16 CEST
UCS 5.0 has been released:
 https://docs.software-univention.de/release-notes-5.0-0-en.html
 https://docs.software-univention.de/release-notes-5.0-0-de.html

If this error occurs again, please use "Clone This Bug".