Bug 39024 - Update of 3.2 UEFI to 4.0 makes system unbootable
Update of 3.2 UEFI to 4.0 makes system unbootable
Status: RESOLVED WORKSFORME
Product: UCS
Classification: Unclassified
Component: Kernel
UCS 4.0
Other Linux
: P5 normal (vote)
: ---
Assigned To: Kernel maintainers
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2015-07-26 13:00 CEST by Tim Petersen
Modified: 2017-04-21 12:20 CEST (History)
5 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 6: Setup Problem: Issue for the setup process
Who will be affected by this bug?: 1: Will affect a very few installed domains
How will those affected feel about the bug?: 2: A Pain – users won’t like this once they notice it
User Pain: 0.069
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Ticket number: 2015072621000069 2015072221000236
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Petersen univentionstaff 2015-07-26 13:00:44 CEST
+++ This bug was initially created as a clone of Bug #36335 +++

We need at least:

CONFIG_MODULE_SIG=y
CONFIG_MODULE_SIG_SHA512=y
CONFIG_MODULE_SIG_HASH="sha512"
CONFIG_MODULE_SIG_ALL=y

CONFIG_MODULE_SIG_FORCE needs to be checked.
--------------------------------------------------------


2015072621000069
2015072221000236


I think we have a problem here, as three customers reported "invalid signature" failures in grub after reboot.



Fehler: /vmlinuz-3.16.0-ucs135-amd64.efi.signed has invalid signature
Fehler: Sie müssen zuerst den Kernel laden.

Beliebiege Taste um fortzusetzen...
Comment 1 Stefan Gohmann univentionstaff 2015-07-26 19:29:06 CEST
Bug #36335 was about module signing. At least at 2015072621000069 the kernel can't be loaded.

Has grub been updated and is it used?
Comment 2 Janek Walkenhorst univentionstaff 2015-07-27 18:52:01 CEST
When a system was installed via the 3.2 UEFI DVD the update to 4.0 makes the system unbootable with the message:

Fehler: /vmlinuz-3.16.0-ucs135-amd64.efi.signed has invalid signature
Fehler: Sie müssen zuerst den Kernel laden.

The second message is just the initrdefi command failing because the linuxefi command failed.

The first message is the direct symptom of the broken boot configuration after the update:
The UEFI support in UCS 4.0 requires signature validation because the grub.cfg is generated with the commands linuxefi/initrdefi (instead of linux/initrd)
For this, GRUB requires the shim (-signed?) package for signature validation.
For booting via the shim, it requires the signed GRUB package (grub-efi-amd64-signed)

Both shim-signed and grub-efi-amd64-signed are not required by any other packages. They should be a (amd64 only?) dependency of univention-grub.

Additionally just installing both packages does not fix the problem, parlty the EFI-directory/-partition is not correctly detected:
On 3.2 systems it is in /boot/grub(/EFI), on 4.0 systems it is in /boot/efi(/EFI)

* Workaround for no boot due to ".efi.signed has invalid signature"
Press "e" to edit the boot entry, modify the "linuxefi" respectively "initrdefi" commands to be just "linux" respectively "initrd", press F10 to boot.

See also Bug #39027
Comment 3 Sönke Schwardt-Krummrich univentionstaff 2015-07-28 08:48:24 CEST
So broken systems can be fixed by manually altering the grub config once to boot into the UCS system and then install the packages shim-signed and grub-efi-amd64-signed? Is anything else required? grub-install? grub-update?

Possible fix:
Maybe we can alter the postup.sh of UCS 4.0. If it detects that the system is in EFI mode, the two packages are installed too.
Comment 4 Tim Petersen univentionstaff 2015-07-28 08:49:28 CEST
The following steps fixed it in the Forum:
- /boot nach /boot.org kopiert
- in /boot die neuen Kernelimages gelöscht
- die /boot/grub/grub.cfg mit grub-mkconfig -o /boot/grub/grub.cfg überschrieben
- reboot
Comment 5 Tim Petersen univentionstaff 2015-07-28 08:50:07 CEST
(In reply to Tim Petersen from comment #4)
> The following steps fixed it in the Forum:
> - /boot nach /boot.org kopiert
> - in /boot die neuen Kernelimages gelöscht
> - die /boot/grub/grub.cfg mit grub-mkconfig -o /boot/grub/grub.cfg
> überschrieben
> - reboot

ergh - not "fixed" - workarounded, of course
Comment 6 Janek Walkenhorst univentionstaff 2015-07-28 11:30:10 CEST
(In reply to Sönke Schwardt-Krummrich from comment #3)
> So broken systems can be fixed by manually altering the grub config once to
> boot into the UCS system
Yes
> and then install the packages shim-signed and
> grub-efi-amd64-signed? Is anything else required? grub-install? grub-update?
The current work-in-progress state of the fix is:

 install shim-signed, grub-efi-amd64-signed
 execute grub-install --efi-directory /boot/grub

But this only leads to a completely unbootable system.

> Possible fix:
> Maybe we can alter the postup.sh of UCS 4.0. If it detects that the system
> is in EFI mode, the two packages are installed too.
Sounds like a reasonable part of the fix.
Comment 7 Jens Thorp-Hansen univentionstaff 2015-08-05 13:06:51 CEST
(In reply to Tim Petersen from comment #4)
> The following steps fixed it in the Forum:
> - /boot nach /boot.org kopiert
> - in /boot die neuen Kernelimages gelöscht
> - die /boot/grub/grub.cfg mit grub-mkconfig -o /boot/grub/grub.cfg
> überschrieben
> - reboot

Note: workaround will probably work only till the next Grub/Kernel update.
Comment 8 Janek Walkenhorst univentionstaff 2015-08-05 18:53:06 CEST
Doing the following after the upgrade to UCS 4.0 (4.0-2 (errata254)) before the reboot fixed the UEFI/grub-configuration on the test hardware:

With aptitude for manual conflict resolution:
  Install shim-signed grub-efi-amd64-signed # install missing software
  Remove delo # remove useless bootloader
umount /boot/grub # (temporarily) umount EFI partition
$EDITOR /etc/fstab # modify /boot/grub entry to be mounted at /boot/efi
mkdir /boot/efi # create new mountpoint
mount /boot/efi # remount EFI partition (on new mountpoint)
find /boot/efi/ -mindepth 1 -xdev -delete # clean EFI partition
grub-install # reinstall grub
update-grub # make sure the config is up-to-date


On reboot the boot failed with ".efi.signed has invalid signature" - switching the BIOS to SecureBoot enabled fixed this problem and the system booted fine.

This (being the same error message) leads to the question if simply switching SecureBoot on fixes this problem without any other changes. (Because of missing shim-signed, this seems unlikely)

It is however surprising that grub would require validation when SecureBoot is not active.
Comment 9 Janis Meybohm univentionstaff 2015-08-10 09:47:55 CEST
Did not work for at least one customer, please see http://forum.univention.de/viewtopic.php?f=48&t=4215
Comment 10 Stefan Gohmann univentionstaff 2015-09-07 06:57:07 CEST
Please have a look. I'm not sure if the new signed grub efi package is part of the 4.0-3 DVD.
Comment 11 Sönke Schwardt-Krummrich univentionstaff 2015-09-21 13:26:58 CEST
(In reply to Janis Meybohm from comment #9)
> Did not work for at least one customer, please see
> http://forum.univention.de/viewtopic.php?f=48&t=4215

Customer booted system with SuperGRUB. I'm suspecting that the system has not been started in EFI mode but in BIOS mode. Therefore grub is looking for different binaries during grub-install/package installation that cannot be found:

source_dir existiert nicht. Bitte geben Sie --target oder --directory an

Despite that, the current 4.0-3-ISO is broken for EFI. See bug 39367.

I was able to repair "unbootable" systems that have been updated from 3.2-x to 4.0-0 by following Janeks steps in comment 8. If secure boot is NOT used, the system is still bootable. The user has to boot into GRUB, edit the first boot entry by pressing "e" and alter two keywords:
"linuxefi" → "linux" and "initrdefi" → "initrd". Then boot the system by pressing F10 or Strg-X.

This disables the signature check for kernel and initrd.

Reason: In 3.2-X there was no shim and during the update to 4.0, the update of the EFI configuration fails telling EFI that the shim has to be started instead of GRUB.
GRUB uses shim to verify signatures of binary files. If shim has not been started, the verification fails.

After performing Janeks steps of comment 8, shim is configured as default start binary in EFI. This can be checked by calling "efibootmgr -v". Check default boot order and the boot entries. First default boot entry should contain the shim.efi binary.

If this is not the case, please try the following commands in that order. 
PLEASE NOTE: do *NOT* call the find command on dual boot EFI systems. Otherwise the other OS may not boot anymore!
# find /boot/efi/ -mindepth 1 -xdev -delete # clean EFI partition
# univention-install --reinstall shim-signed grub-efi-amd64-signed
# grub-install
# update-grub

Do we need an SDB article for this issue?
Otherwise I would set this bug to RESOLVED.

(In reply to Stefan Gohmann from comment #10)
> Please have a look. I'm not sure if the new signed grub efi package is part
> of the 4.0-3 DVD.

This problem is handled via bug 39367.
Comment 12 Sönke Schwardt-Krummrich univentionstaff 2016-02-24 12:31:15 CET
The workaround has been documented in http://sdb.univention.de/1359
Comment 13 Philipp Hahn univentionstaff 2017-04-21 12:20:34 CEST
UCS-3.2 is OoM.
An SDB entry exists describing the work-around.
Not reported again.