Bug 38307 - External IP routing does not work for docker guests if univention-firewall is restarted
External IP routing does not work for docker guests if univention-firewall is...
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Docker
UCS 4.0
Other Linux
: P5 normal (vote)
: UCS 4.1
Assigned To: Sönke Schwardt-Krummrich
Daniel Tröder
: interim-2
Depends on:
Blocks: 39686 39773
  Show dependency treegraph
 
Reported: 2015-04-21 11:08 CEST by Sönke Schwardt-Krummrich
Modified: 2016-09-16 09:41 CEST (History)
4 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sönke Schwardt-Krummrich univentionstaff 2015-04-21 11:08:09 CEST
BEFORE:

root@docker51:~# iptables -L -n -v -t nat ; iptables -L -n -v
Chain PREROUTING (policy ACCEPT 12 packets, 1702 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    4   240 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 6 packets, 520 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 1 packets, 60 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    1    60 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 1 packets, 60 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
Chain INPUT (policy ACCEPT 367 packets, 25119 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 403 packets, 47137 bytes)
 pkts bytes target     prot opt in     out     source               destination         
root@docker51:~# 

FIREWALL SHUTDOWN:
root@docker51:~# invoke-rc.d univention-firewall stop
Stopping Univention iptables configuration::.
root@docker51:~#

AFTER:
root@docker51:~# iptables -L -n -v -t nat ; iptables -L -n -v
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (0 references)
 pkts bytes target     prot opt in     out     source               destination         
Chain INPUT (policy ACCEPT 28 packets, 1576 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 29 packets, 3404 bytes)
 pkts bytes target     prot opt in     out     source               destination         
root@docker51:~# 


The same problem applies to a firewall restart/start.
Comment 1 Arvid Requate univentionstaff 2015-08-19 20:12:24 CEST
Possibly this is useful, at least good to know:

> "Docker will not delete or modify any pre-existing rules from the DOCKER filter chain. This allows the user to create in advance any rules required to further restrict access to the containers."
                             -- https://docs.docker.com/articles/networking/
Comment 2 Daniel Tröder univentionstaff 2015-08-28 11:45:15 CEST
There is a feature request for docker to export its iptables rules: https://github.com/docker/docker/issues/12791 from which a shell script was cerated that does it: https://github.com/icy/hsabymho/blob/master/libs/docker.sh

More stable is probably to use the Python-API to retrieve the information from "docker inspect": http://docker-py.readthedocs.org/en/latest/api/#inspect_container and create the rules from that as a step in 10_univention-firewall_start.sh
Comment 3 Daniel Tröder univentionstaff 2015-09-15 16:07:29 CEST
Added /etc/security/packetfilter.d/20_docker.sh that runs only if docker is installed and UCS is not inside a container.
It creates the exact same rules docker creates (checked with iptables-save).
It assumes the bridge is called "docker0" and the forwarded ports should appear on all external interfaces (both default).

Commits: 63737 (copyright), 63738 (code)
Comment 4 Arvid Requate univentionstaff 2015-09-30 18:22:36 CEST
Package version 7.0.0-4 has not been imported+built. Please add an entry to doc/changelog/changelog-4.1-0.xml too.
Comment 5 Daniel Tröder univentionstaff 2015-10-01 09:53:22 CEST
Created a changelog entry in commit 64125 imported to 4.1-0 and built package (7.0.0-4.81.201510010951).
Comment 6 Arvid Requate univentionstaff 2015-10-01 18:36:57 CEST
Ok, I just added a small fix for:

============================================================================
root@xen1:~# /etc/init.d/docker stop
[ ok ] Stopping Docker: docker.
root@xen1:~# /etc/init.d/univention-firewall restart
[ ok ] Stopping Univention iptables configuration::.
[....] Starting Univention iptables configuration::time="2015-10-01T11:28:35-04:00" level=fatal msg="Get http:///var/run/docker.sock/v1.18/containers/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?" 
. ok 
============================================================================

debian/changelog message for that:

  * Check if docker daemon runs before sending commands (Bug #38307)

Otherwise seems to work.
Comment 7 Felix Botner univentionstaff 2015-10-28 16:58:19 CET
(In reply to Daniel Tröder from comment #5)
> (IMO this kind of belongs to #38307, but I fixed it referencing this bug.)
> 
> I moved the Docker container rules to the DOCKER chain, so the Docker engine
> removes them when it shuts down containers.
> 
> Commit: 64626

see Bug #39429

The docker rules are still moved to the FORWARD chain by univention-firewall restart. Is that OK?

-> /etc/init.d/docker start
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain DOCKER (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.17.0.1           tcp dpt:https
ACCEPT     tcp  --  anywhere             172.17.0.1           tcp dpt:http

->/etc/init.d/univention-firewall restart
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             172.17.0.1           tcp dpt:https
ACCEPT     tcp  --  anywhere             172.17.0.1           tcp dpt:http

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Chain DOCKER (0 references)
target     prot opt source               destination
Comment 8 Arvid Requate univentionstaff 2015-10-28 17:25:28 CET
Hmm, and I think there is a second issue: When univention-firewall is stopped, the containers loose their connection:


=============================================================
root@master60:~# docker exec -it fff8bf71a5b5 ping -c1 -W3 www.google.de
PING www.google.de (173.194.112.31) 56(84) bytes of data.
64 bytes from fra07s27-in-f31.1e100.net (173.194.112.31): icmp_req=1 ttl=49 time=12.1 ms

--- www.google.de ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 12.154/12.154/12.154/0.000 ms

root@master60:~# /etc/init.d/univention-firewall stop
Stopping Univention iptables configuration::.

root@master60:~# docker exec -it fff8bf71a5b5 ping -c1 -W3 www.google.de
PING www.google.de (173.194.112.23) 56(84) bytes of data.
root@master60:~#
=============================================================
Comment 9 Sönke Schwardt-Krummrich univentionstaff 2015-10-30 13:57:43 CET
(In reply to Arvid Requate from comment #8)
> Hmm, and I think there is a second issue: When univention-firewall is
> stopped, the containers loose their connection:

For UCS 4.1-0 we will keep the current status. This issue is addressed later on in a separate bug.

If "univention-firewall" is stopped, all containers will loose connectivity until "docker" is restarted or the system is rebooted (if univention-firewall is deactivated permanently via security/packetfilter/disabled=yes).

(In reply to Felix Botner from comment #7)
> The docker rules are still moved to the FORWARD chain by univention-firewall
> restart. Is that OK?

Fixed. Additionally the DOCKER queue is created if missing.

univention-firewall (7.0.0-8):
r65007 | Bug #38307: create DOCKER queue if missing / simplified code

Successful build
Package: univention-firewall
Version: 7.0.0-8.84.201510301351
User: sschwardt
Branch: ucs_4.1-0
Comment 10 Daniel Tröder univentionstaff 2015-11-05 14:11:04 CET
OK: univention-firewall restart recreates the same rules are before
OK: docker removes rules when it shuts down a container

Tests:

1. boot with no docker containers, with docker and univention-firewall started
2. start a container (dudle-docker)

# iptables -L -n -v

Chain INPUT (policy DROP 0 packets, 0 bytes)
[no rule changed]
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
[no rule changed]
Chain OUTPUT (policy ACCEPT 289 packets, 70180 bytes)
[no rule changed]
Chain DOCKER (1 references)
+    0     0 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.1           tcp dpt:443
+    0     0 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.1           tcp dpt:80

# iptables -L -n -v -t nat

Chain POSTROUTING (policy ACCEPT 3 packets, 442 bytes)
+    0     0 MASQUERADE  tcp  --  *      *       172.17.0.1           172.17.0.1           tcp dpt:443
+    0     0 MASQUERADE  tcp  --  *      *       172.17.0.1           172.17.0.1           tcp dpt:80

Chain DOCKER (2 references)
+    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:40001 to:172.17.0.1:443
+    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:40000 to:172.17.0.1:80

3. restart univention-firewall

# /etc/init.d/univention-firewall restart
# iptables -L -n -v

Chain INPUT (policy DROP 0 packets, 0 bytes)
[no rule changed]
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
[no rule changed]
Chain OUTPUT (policy ACCEPT 289 packets, 70180 bytes)
[no rule changed]
Chain DOCKER (1 references)
[no rule changed]

# iptables -L -n -v -t nat

Chain POSTROUTING (policy ACCEPT 23 packets, 1380 bytes)
[no rule changed]
Chain DOCKER (2 references)
[no rule changed]

4. stop and start container

#/etc/init.d/docker-app-dudle-docker stop
#/etc/init.d/docker-app-dudle-docker start

# iptables -L -n -v

Chain INPUT (policy DROP 0 packets, 0 bytes)
[no rule changed]
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
[no rule changed]
Chain OUTPUT (policy ACCEPT 289 packets, 70180 bytes)
[no rule changed]
Chain DOCKER (1 references)
[no rule changed, only IP of container changed]

# iptables -L -n -v -t nat

Chain POSTROUTING (policy ACCEPT 23 packets, 1380 bytes)
[no rule changed, only IP of container changed]
Chain DOCKER (2 references)
[no rule changed, only IP of container changed]

5. restart univention-firewall

# /etc/init.d/univention-firewall restart
# iptables -L -n -v

Chain INPUT (policy DROP 0 packets, 0 bytes)
[no rule changed]
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
[no rule changed]
Chain OUTPUT (policy ACCEPT 289 packets, 70180 bytes)
[no rule changed]
Chain DOCKER (1 references)
[no rule changed]

# iptables -L -n -v -t nat

Chain POSTROUTING (policy ACCEPT 23 packets, 1380 bytes)
[no rule changed]
Chain DOCKER (2 references)
[no rule changed]

6. stop container → docker removes rules

# /etc/init.d/docker-app-dudle-docker stop
# iptables -L -n -v

Chain INPUT (policy DROP 0 packets, 0 bytes)
[no rule changed]
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
[no rule changed]
Chain OUTPUT (policy ACCEPT 289 packets, 70180 bytes)
[no rule changed]
Chain DOCKER (1 references)
-    0     0 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.3           tcp dpt:443
-    0     0 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.3           tcp dpt:80

# iptables -L -n -v -t nat

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
-    0     0 MASQUERADE  tcp  --  *      *       172.17.0.3           172.17.0.3           tcp dpt:443
-    0     0 MASQUERADE  tcp  --  *      *       172.17.0.3           172.17.0.3           tcp dpt:80

Chain DOCKER (2 references)
-    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:40001 to:172.17.0.3:443
-    0     0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:40000 to:172.17.0.3:80


-------------------------------------------------------------------

(In reply to Sönke Schwardt-Krummrich from comment #9)
> (In reply to Arvid Requate from comment #8)
> > Hmm, and I think there is a second issue: When univention-firewall is
> > stopped, the containers loose their connection:
> For UCS 4.1-0 we will keep the current status. This issue is addressed later
> on in a separate bug.Bug #39686
Comment 11 Stefan Gohmann univentionstaff 2015-11-17 12:11:51 CET
UCS 4.1 has been released:
 https://docs.software-univention.de/release-notes-4.1-0-en.html
 https://docs.software-univention.de/release-notes-4.1-0-de.html

If this error occurs again, please use "Clone This Bug".
Comment 12 Daniel Tröder univentionstaff 2016-09-16 09:41:27 CEST
Gah... docker dev (although not many commits) dismisses the non-manageability of dockers iptables handling: https://github.com/docker/docker/issues/12791