Bug 39126 - Upgrade to UCS 4.0-3 failed in jenkins due to linux-image-3.16.0-ucs135-amd64
Upgrade to UCS 4.0-3 failed in jenkins due to linux-image-3.16.0-ucs135-amd64
Status: RESOLVED WONTFIX
Product: UCS Test
Classification: Unclassified
Component: General
unspecified
Other Linux
: P5 normal (vote)
: ---
Assigned To: UCS maintainers
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-10 07:52 CEST by Stefan Gohmann
Modified: 2021-12-10 11:06 CET (History)
3 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments
Fix univention-grub (5.94 KB, patch)
2015-08-11 17:26 CEST, Philipp Hahn
Details | Diff
updater.log (1.11 MB, text/plain)
2016-06-09 22:17 CEST, Stefan Gohmann
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Gohmann univentionstaff 2015-08-10 07:52:37 CEST
From time to time the upgrade failed in Jenkins:

Setting up linux-image-3.16.0-ucs135-amd64 (3.16.7-ckt11-1~bpo70+1.135.201507161851) ...
Running /usr/sbin/update-grub.
Generating grub.cfg ...
Found background: /boot/grub/uniboot.png
Found background image: /boot/grub/uniboot.png
Found linux image: /boot/vmlinuz-3.16.0-ucs135-amd64
Found linux image: /boot/vmlinuz-3.16-ucs109-amd64
Found initrd image: /boot/initrd.img-3.16-ucs109-amd64
Found memtest86+ image: /boot/memtest86+.bin
  No volume groups found
done
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-3.16.0-ucs135-amd64
/etc/kernel/postinst.d/zz-update-grub:
Generating grub.cfg ...
Found background: /boot/grub/uniboot.png
Found background image: /boot/grub/uniboot.png
Found linux image: /boot/vmlinuz-3.16.0-ucs135-amd64
Found initrd image: /boot/initrd.img-3.16.0-ucs135-amd64
Found linux image: /boot/vmlinuz-3.16-ucs109-amd64
Found initrd image: /boot/initrd.img-3.16-ucs109-amd64
Found memtest86+ image: /boot/memtest86+.bin
  No volume groups found
done
dpkg: error processing linux-image-3.16.0-ucs135-amd64 (--configure):
 subprocess installed post-installation script returned error exit status 128
Setting up univention-mail-postfix (9.0.3-4.260.201507231625) ...
Comment 1 Stefan Gohmann univentionstaff 2015-08-10 07:53:42 CEST
Please have a look at 10.210.123.131. The system is still up and running.
Comment 2 Philipp Hahn univentionstaff 2015-08-10 18:41:51 CEST
The "univention-grub" upgrade is not yet configured, while the new kernel is installed:

13:22:58 status unpacked univention-grub:all 8.0.4-2.124.201506300919
...
13:23:06 configure linux-image-3.16.0-ucs135-amd64:amd64 3.16.7-ckt11-1~bpo70+1.135.201507161851 <none>
13:23:06 status unpacked linux-image-3.16.0-ucs135-amd64:amd64 3.16.7-ckt11-1~bpo70+1.135.201507161851
13:23:06 status half-configured linux-image-3.16.0-ucs135-amd64:amd64 3.16.7-ckt11-1~bpo70+1.135.201507161851
...
13:23:49 configure univention-grub:all 8.0.4-2.124.201506300919 <none>

The kernel executes "/etc/kernel/postinst.d/zz-update-grub",
which exec's "update-grub",
which calls "/usr/sbin/univention-grub-generate-menu-lst" as its end,
which sets the exit status of the command.


Please note the double-invocation in "update-grub" in univention/updater.log:
> Setting up grub-pc (2.00-18.92.201506300903) ...
> Removing update-grub hooks from /etc/kernel-img.conf in favour of /etc/kernel/ hooks.

$ ucr filter </etc/univention/templates/files/etc/kernel-img.conf | grep ^postinst_hook
postinst_hook = /usr/sbin/update-grub

TODO: Remove the hook from our UCR template.
Comment 3 Philipp Hahn univentionstaff 2015-08-11 17:26:21 CEST
(In reply to Philipp Hahn from comment #2)
...
> The kernel executes "/etc/kernel/postinst.d/zz-update-grub",
> which exec's "update-grub",
> which calls "/usr/sbin/univention-grub-generate-menu-lst" as its end,
> which sets the exit status of the command.

This would print a different error message, if any of the hook scripts fail.


> Please note the double-invocation in "update-grub" in univention/updater.log:
> > Setting up grub-pc (2.00-18.92.201506300903) ...
> > Removing update-grub hooks from /etc/kernel-img.conf in favour of /etc/kernel/ hooks.

See attachment for a patch.


Further analysis:

- /var/lib/dpkg/info/linux-image-3.16.0-ucs135-amd64.postinst is a PERL script

- after executing the the hook scripts, is exit(0)s.

- it uses debconf!

- the following error message indicates, that the PERL interpreter exited:
> dpkg: error processing linux-image-3.16.0-ucs135-amd64 (--configure):
>  subprocess installed post-installation script returned error exit status 128

- there are several public bug reports, where dpkg returns 128.

- it might be related to debconf getting confused on exit: <https://bugs.launchpad.net/ubuntu/+source/debconf/+bug/996712>
<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=204054>

- in EC2 GRUB also uses debconf to ask a question about the missing /dev/xdva1 mapping.

- This error message is from "lvm2", probably related to "/dev/xvda1":
> No volume groups found


Will wait for next occurrence.
Comment 4 Philipp Hahn univentionstaff 2015-08-11 17:26:50 CEST
Created attachment 7098 [details]
Fix univention-grub
Comment 5 Florian Best univentionstaff 2016-02-08 14:04:27 CET
This problem hit me, too.
Comment 7 Stefan Gohmann univentionstaff 2016-06-09 22:17:04 CEST
Created attachment 7731 [details]
updater.log
Comment 8 Stefan Gohmann univentionstaff 2016-08-19 13:24:37 CEST
It seems to work after removing this line from the Jenkins EC2 test setup:

 echo set grub-pc/install_devices_empty true | debconf-communicate

At least the UCS@school upgrade tests work again: r71726
Comment 9 Janek Walkenhorst univentionstaff 2016-08-25 14:12:52 CEST
It seems that the failure occurs during any of
 /etc/grub.d/20_memtest86+
 /etc/grub.d/30_os-prober
 /etc/grub.d/30_uefi-firmware
 /etc/grub.d/40_custom
 /etc/grub.d/41_custom
and it then bubbles up the stack of
 /usr/sbin/grub-mkconfig
 /usr/sbin/update-grub
 /etc/kernel/postinst.d/zz-update-grub
 /var/lib/dpkg/info/linux-image-4.1-0-ucs190-amd64.postinst

The ami-f2934981 (Univention Corporate Server (UCS) 4.0 (official image) rev. 7) has /dev/vda instead of /dev/xvda configured as grub-pc/install_devices but the workaround in utils.sh seems to correct it.
In any case I don't expect the grub-pc/install_devices to be the problem.
Comment 10 Philipp Hahn univentionstaff 2021-12-10 11:06:40 CET
UCS<<4.4 is EOL → WONTFIX