Univention Bugzilla – Bug 50261
Fix update to UCS 4.4-2 issue with docker packages
Last modified: 2019-11-19 13:16:43 CET
At bug #50259 we blocked the update to UCS 4.4-2 due to an issue with the docker packages. In https://help.univention.com/t/13088 and Ticket#2019092421000927 the update to UCS 4.4-2 fails. At this bug a fix will be developed. The preup block should be removed as well.
Information about the initial UCS Version and installed Apps can be found on Ticket#2019092421000927.
I just upgraded a UCS-4.0 system to 4.4-2; UCS-4.0 uses a 3.16 kernel, which did not yet have "overlayfs", but used "aufs" from Debian. Only with UCS-4.1 we switched to linux-4.1, which supports "overlay". The default was changed with Bug #40515. So any system setup before UCS-4.1 should be affected.
We should fix this issue quite soon to allow administrators to upgrade to 4.4-2, because according to our maintenance life cycle 4.4-1 will stop to receive updates at beginning of November 2019. See https://help.univention.com/t//13088/10
Suggested approach: check "docker info" or "ucr get docker/daemon/default/opts/storage-driver" if != "overlay" cancel update mv /var/lib/docker/aufs /var/lib/docker/aufs.bak mv /var/lib/docker/devicemapper /var/lib/docker/devicemapper.bak proceed with univention-upgrade
(In reply to Dirk Wiesenthal from comment #4) > Suggested approach: > > check "docker info" or "ucr get docker/daemon/default/opts/storage-driver" > > if != "overlay" > cancel update > > mv /var/lib/docker/aufs /var/lib/docker/aufs.bak [ -d /var/lib/docker/aufs ] && find /var/lib/docker/aufs -type d -empty -delete [ -d /var/lib/docker/aufs ] && mv /var/lib/docker/aufs /var/lib/docker/aufs.moved-by-ucs4.4-2-upgrade > mv /var/lib/docker/devicemapper /var/lib/docker/devicemapper.bak dito? > proceed with univention-upgrade
585c94ec64 Bug #50261: check current docker storage driver and abort upgrade if not 'overlay' preup.sh uploaded to test_mirror
Works as expected. REOPENED for minor wording issues and copying it to the live server eventually
dfe8bfa222 Bug #50261: adjust error message for wrong docker storage driver no changelog/build needed copied preup.sh and preup.sh.gpg to /var/univention/buildsystem2/mirror/ftp/4.4/maintained/4.4-2/all/
Update from 4.0 to 4.4 w/ App: OK Update from 4.0 to 4.4 w/o App: OK Update from 4.0 to 4.4 w/ wrong storage driver: OK Update from 4.0 to 4.4 w/o running docker service: OK Update from 4.3 to 4.4 w/ App: OK Update from 4.3 to 4.4 w/o App: OK Wording: OK
3125ad036a Bug #50261: link to sdb article copied to {test_}mirror synced to extern SDB article https://help.univention.com/t/problem-upgrade-to-ucs-4-4-2-is-blocked-due-to-docker-storage/13368
With all due respect, that support article is lacking on important details: > Changing the storage driver causes docker to not find previously created containers anymore since the location where container data is stored changed. To keep them they have to be migrated. _How_ are users supposed to migrate the containers? At the moment this sounds like "well, your installed Docker apps will be buggered; tough luck". That's not a solution.
Hi While the applied fixes and docs should now allow updating docker-free systems from 4.4-1 to 4.4-2, I do share the concerns expressed by Moritz. The workaround procedure should definitely show how to migrate the docker containers and be a bit more explicit on how it impact dockerized applications. Also this is issue is something that should should be part of an update to the release notes of 4.4-2. System administrators and service provides do actually read them and use them for preparation during release upgrades. Also please consider sharing an update on how this blocker affects the end of maintenance for 4.4-1. -- Mathieu
(In reply to Moritz Bunkus from comment #11) > _How_ are users supposed to migrate the containers? At the moment this > sounds like "well, your installed Docker apps will be buggered; tough luck". > That's not a solution. (In reply to Mathieu Simon from comment #12) > Hi > > While the applied fixes and docs should now allow updating docker-free > systems from 4.4-1 to 4.4-2, I do share the concerns expressed by Moritz. > The workaround procedure should definitely show how to migrate the docker > containers and be a bit more explicit on how it impact dockerized > applications. We released this fix in the preup.sh so that users can now update to 4.4-2 where it was blocked before. The accompanying article does not provide an extensive solution. The link to the article is shown in UCS if the docker daemon of the 4.4-1 system does have a configured storage driver other than "overlay". We assumed the number of systems to be rather low compared to the systems that were blocked from the update prior to this change. May I ask, is any of your systems affected? If so, what storage driver do you / did you use? Do you use any App Center Apps? Which other Docker Containers are running? As any "untouched" UCS 4.4-1 uses overlay, any other scenario should involve some kind of "custom setup". While we do allow admins to customize their systems, of course, we are reluctant to offer a "one size fits all" migration guide for these setups. We will gladly update the current article with our findings if you could provide us some insights in what you are using. Migrating the containers should not be too difficult. Stopping them and reinitializing them after the daemon restart should be enough (needs re-pulling). It is the data that may be problematic. Did you use a volume (-v myvolume:/data)? Then everything should be in place after the service restart. But as we do not know your setup, we have not yet written anything in the article. > > Also this is issue is something that should should be part of an update to > the release notes of 4.4-2. System administrators and service provides do > actually read them and use them for preparation during release upgrades. > That is a good point. We will talk about this. > Also please consider sharing an update on how this blocker affects the end > of maintenance for 4.4-1. > We are currently discussing how to approach this, yes.
Hi Dirk Not speaking for Moritz but only for my use case: > We released this fix in the preup.sh so that users can now update to 4.4-2 > where it was blocked before. The accompanying article does not provide an > extensive solution. That's fine. > The link to the article is shown in UCS if the docker daemon of the 4.4-1 > system does have a configured storage driver other than "overlay". > > We assumed the number of systems to be rather low compared to the systems > that were blocked from the update prior to this change. It could be useful to check with your statistics you get via the free users, but it may not be representative. > May I ask, is any of your systems affected? If so, what storage driver do > you / did you use? Do you use any App Center Apps? Which other Docker > Containers are running? Our systems all use the default overlay storage driver so we're not affected. If overlay was and is the supported and expected configuration, then this should be outlined in the article. Our systems were updated since the early 2.4 days but they have all the same settings. I think the point is to give some reassuring words. If i.e. the git history shows that overlay was always the default, highligt this. Maybe for enterprise customers something along the line of the following could be helpful: "the default storage driver is and has been "overlay", if in some rare circumstances you've changed it to something different, please contact Univention support for further assistance". I know plenty examples with another large open source vendor whose KB mentions similar wordings for rare and specific issues that may need case-by-case decision and hence the recommendation is to log a support case referencing the given KB article. Then they ask you for specific environment information and a plan to action is formulated. Also thanks for your update concerning updating the release notes EoL of 4.4-1. Regards Mathieu
Released updated preup.sh
FYI: Because of this update issue and the necessary time to fix it, we have decided to postpone the end of security maintenance for UCS 4.4-1 until 03. Dec. 2019.