Bug 44033 - UCS 4.2: Running "service docker restart" leaves all containers terminated
UCS 4.2: Running "service docker restart" leaves all containers terminated
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Docker
UCS 4.2
Other Linux
: P5 normal (vote)
: UCS 4.2
Assigned To: Stefan Gohmann
Felix Botner
: interim-4
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2017-03-22 18:56 CET by Arvid Requate
Modified: 2017-04-04 18:28 CEST (History)
3 users (show)

See Also:
What kind of report is it?: Release Management
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Arvid Requate univentionstaff 2017-03-22 18:56:15 CET
Running "service docker restart" leaves all containers terminated.

univention-docker ships a template for /etc/init.d/docker to restart running containers, but /lib/systemd/system/docker.service takes precedence.


Maybe we can add this to the /etc/default/docker template:

https://docs.docker.com/engine/admin/live-restore/#enable-the-live-restore-option


We should also check if there are other changes in our init script that need to be migrated.
Comment 1 Stefan Gohmann univentionstaff 2017-03-23 07:27:08 CET
We have the following three special cases in our init script:

1. Linux Kernel: You need at least Kernel 4.1 to use it. I think we can skip this test.

2. Network conflict: If the docker default network overlaps with the default network, a log message has been shown.

3. The Firewall is restated.

I guess we can add system diagnostic scripts for the first two issues if they are still relevant.

The firewall restart has been added due to Bug #42698. Currently, I don't see why these changes are needed. Unfortunately, the bug doesn't describe the impact.

So, I would recommend to add a new UCR variable which allows to define docker parameters and remove the old init script.
Comment 2 Stefan Gohmann univentionstaff 2017-03-23 11:08:40 CET
(In reply to Stefan Gohmann from comment #1)
> The firewall restart has been added due to Bug #42698. Currently, I don't
> see why these changes are needed. Unfortunately, the bug doesn't describe
> the impact.

OK, the reason is that we need some more firewall rules for example the MySQL settings. We could restart univention-firewall via ExecStartPos in the docker service.
Comment 3 Stefan Gohmann univentionstaff 2017-03-23 17:53:11 CET
r78226: univention-firewall 
* 20_docker.sh: Use 'pidof dockerd' instead of systemctl since the
  script is executed through systemd (Bug #44033)

r78196:
* Remove UCR template for /etc/init.d/docker. The systemd service will
  be used (Bug #44033)
* Added a new UCR variable docker/daemon/default/parameter/.* to add
  additional parameter for the Docker daemon (Bug #44033)
* Activate live-restore by default (Bug #44033)


r78227: Changelog
Comment 4 Felix Botner univentionstaff 2017-03-24 10:59:58 CET
/etc/init.d/docker still exists after the update (/etc/univention/templates/files/etc/init.d/docker has been removed).

Sometimes after the restart a ssh login is not possible. Unfortunately, this happens not always. Strangely, iptables seems to be correct

ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh

but a ssh login does not work until i restart the firewall.
Comment 5 Stefan Gohmann univentionstaff 2017-03-24 15:37:28 CET
r17428:
* Ensure that firewall init script has been finished (Bug #44033)
Comment 6 Felix Botner univentionstaff 2017-03-24 16:42:04 CET
Now i had the situation that notifier port was blocked (and the join of other servers failed), after restarting the firewall, the notifier was reachable.
Comment 7 Stefan Gohmann univentionstaff 2017-03-24 19:23:35 CET
(In reply to Felix Botner from comment #6)
> Now i had the situation that notifier port was blocked (and the join of
> other servers failed), after restarting the firewall, the notifier was
> reachable.

Can you append the output of journalctl? I'm still unable to reproduce it.
Comment 8 Stefan Gohmann univentionstaff 2017-03-24 19:53:02 CET
(In reply to Stefan Gohmann from comment #7)
> (In reply to Felix Botner from comment #6)
> > Now i had the situation that notifier port was blocked (and the join of
> > other servers failed), after restarting the firewall, the notifier was
> > reachable.
> 
> Can you append the output of journalctl? I'm still unable to reproduce it.

I hope it has been fixed with:

r17429 (docker.io 1.12.6-0ubuntu1~16.04.1A~4.2.0.201703241951):
* Really ensure that firewall init script has been finished (Bug #44033)

Waiting for my tests.
Comment 9 Stefan Gohmann univentionstaff 2017-03-24 20:40:51 CET
(In reply to Stefan Gohmann from comment #8)
> (In reply to Stefan Gohmann from comment #7)
> > (In reply to Felix Botner from comment #6)
> > > Now i had the situation that notifier port was blocked (and the join of
> > > other servers failed), after restarting the firewall, the notifier was
> > > reachable.
> > 
> > Can you append the output of journalctl? I'm still unable to reproduce it.
> 
> I hope it has been fixed with:
> 
> r17429 (docker.io 1.12.6-0ubuntu1~16.04.1A~4.2.0.201703241951):
> * Really ensure that firewall init script has been finished (Bug #44033)
> 
> Waiting for my tests.

My tests were successful. Please try again,
Comment 10 Felix Botner univentionstaff 2017-03-28 16:00:33 CEST
-> docker ps && service docker restart && docker ps 
CONTAINER ID        IMAGE                                                  COMMAND                  CREATED             STATUS              PORTS                                           NAMES
62af6cbcc624        docker.software-univention.de/ucs-appbox-amd64:4.1-3   "/sbin/init"             24 hours ago        Up About a minute   0.0.0.0:40001->80/tcp, 0.0.0.0:40002->443/tcp   awesome_mestorf
ba7b239eb6f4        docker.software-univention.de/nextcloud:11.0.2-0       "/bin/sh -c /usr/sbin"   26 hours ago        Up About a minute   0.0.0.0:40000->80/tcp                           trusting_hypatia
CONTAINER ID        IMAGE                                                  COMMAND                  CREATED             STATUS              PORTS                                           NAMES
62af6cbcc624        docker.software-univention.de/ucs-appbox-amd64:4.1-3   "/sbin/init"             24 hours ago        Up About a minute   0.0.0.0:40001->80/tcp, 0.0.0.0:40002->443/tcp   awesome_mestorf
ba7b239eb6f4        docker.software-univention.de/nextcloud:11.0.2-0       "/bin/sh -c /usr/sbin"   26 hours ago        Up About a minute   0.0.0.0:40000->80/tcp                           trusting_hypatia
Comment 11 Stefan Gohmann univentionstaff 2017-04-04 18:28:20 CEST
UCS 4.2 has been released:
 https://docs.software-univention.de/release-notes-4.2-0-en.html
 https://docs.software-univention.de/release-notes-4.2-0-de.html

If this error occurs again, please use "Clone This Bug".