Bug 57819 - Provide assistance for migrating from docker Storage Driver "overlay" to "overlay2"
Summary: Provide assistance for migrating from docker Storage Driver "overlay" to "ove...
Status: NEW
Alias: None
Product: UCS
Classification: Unclassified
Component: Docker
Version: UCS 5.0
Hardware: Other Linux
: P5 normal
Target Milestone: ---
Assignee: App Center maintainers
QA Contact: App Center maintainers
URL: https://git.knut.univention.de/univen...
Keywords:
Depends on: 40515
Blocks:
  Show dependency treegraph
 
Reported: 2025-01-02 13:54 CET by Arvid Requate
Modified: 2025-01-02 15:14 CET (History)
0 users

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Customer ID:
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Arvid Requate univentionstaff 2025-01-02 13:54:20 CET
Bug 40515 added the UCR variable docker/daemon/default/opts/storage-driver and set it to "overlay" in univention-docker.postinst.

Later, in 4.4-2 we stopped setting this variable, see Bug 50295 Comment 3.

(We also had related issues with the switch from aufs to overlay, see Bug 50261)

In our upgrade tests with systems from UCS 2.x to 5.2 we see that

* There are "situations" with UCS 5.0-9 systems where docker runs with Storage Driver
  overlay2 even though the UCR variable is set to overlay. So docker seems to ignore
  the "--storage-driver=overlay" option set via DOCKER_OPTS in /etc/default/docker. 
  Accordingly it uses /var/lib/docker/overlay2 and ignores /var/lib/docker/overlay.

* After updating to UCS 5.2, docker seems to again use the storage driver specified
  via UCR variable (i.e. DOCKER_OPTS) and accordingly ignores /var/lib/docker/overlay2.
  So all previously running containers are not available any longer.

We'll address the second point via a preup check for 5.1-0 to avoid that incoherent state.

But, there are also UCS 5.0-9 systems that run with "overlay", as specified via UCR variable (i.e. DOCKER_OPTS) and thus use /var/lib/docker/overlay. They also continue to use that deprecated driver after update to UCS 5.2-0.

We should offer a migration path (e.g. a guide) to switch those systems to "overlay2".
Comment 1 Arvid Requate univentionstaff 2025-01-02 15:14:15 CET
Note: I also found a system with 5.0-9 errata1195 that uses "overlay" even though the UCR variable is not set.

In that case journalctl shows a line "[graphdriver] using prior storage driver: overlay",
which seems to get issued when no specific storage driver is specified. In that case

https://github.com/moby/moby/blob/42f40b1d6dd7562342f832b9cd2adf9e668eeb76/daemon/graphdriver/driver.go#L235

and

https://github.com/moby/moby/blob/42f40b1d6dd7562342f832b9cd2adf9e668eeb76/daemon/graphdriver/driver_linux.go#L53

show that docker scans some directories in an ordered priority list, amongst them
/var/lib/docker/overlay2 and later /var/lib/docker/overlay and picks the first that contains subdirectories. In the case of that system, the overlay2 directory existed but was empty (not even the overlay2 specific "linkDir" named "l" was present).