Univention Bugzilla – Bug 38307
External IP routing does not work for docker guests if univention-firewall is restarted
Last modified: 2016-09-16 09:41:27 CEST
BEFORE: root@docker51:~# iptables -L -n -v -t nat ; iptables -L -n -v Chain PREROUTING (policy ACCEPT 12 packets, 1702 bytes) pkts bytes target prot opt in out source destination 4 240 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL Chain INPUT (policy ACCEPT 6 packets, 520 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 1 packets, 60 bytes) pkts bytes target prot opt in out source destination 1 60 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL Chain POSTROUTING (policy ACCEPT 1 packets, 60 bytes) pkts bytes target prot opt in out source destination 0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0 Chain DOCKER (2 references) pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 367 packets, 25119 bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0 Chain OUTPUT (policy ACCEPT 403 packets, 47137 bytes) pkts bytes target prot opt in out source destination root@docker51:~# FIREWALL SHUTDOWN: root@docker51:~# invoke-rc.d univention-firewall stop Stopping Univention iptables configuration::. root@docker51:~# AFTER: root@docker51:~# iptables -L -n -v -t nat ; iptables -L -n -v Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain DOCKER (0 references) pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 28 packets, 1576 bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 29 packets, 3404 bytes) pkts bytes target prot opt in out source destination root@docker51:~# The same problem applies to a firewall restart/start.
Possibly this is useful, at least good to know: > "Docker will not delete or modify any pre-existing rules from the DOCKER filter chain. This allows the user to create in advance any rules required to further restrict access to the containers." -- https://docs.docker.com/articles/networking/
There is a feature request for docker to export its iptables rules: https://github.com/docker/docker/issues/12791 from which a shell script was cerated that does it: https://github.com/icy/hsabymho/blob/master/libs/docker.sh More stable is probably to use the Python-API to retrieve the information from "docker inspect": http://docker-py.readthedocs.org/en/latest/api/#inspect_container and create the rules from that as a step in 10_univention-firewall_start.sh
Added /etc/security/packetfilter.d/20_docker.sh that runs only if docker is installed and UCS is not inside a container. It creates the exact same rules docker creates (checked with iptables-save). It assumes the bridge is called "docker0" and the forwarded ports should appear on all external interfaces (both default). Commits: 63737 (copyright), 63738 (code)
Package version 7.0.0-4 has not been imported+built. Please add an entry to doc/changelog/changelog-4.1-0.xml too.
Created a changelog entry in commit 64125 imported to 4.1-0 and built package (7.0.0-4.81.201510010951).
Ok, I just added a small fix for: ============================================================================ root@xen1:~# /etc/init.d/docker stop [ ok ] Stopping Docker: docker. root@xen1:~# /etc/init.d/univention-firewall restart [ ok ] Stopping Univention iptables configuration::. [....] Starting Univention iptables configuration::time="2015-10-01T11:28:35-04:00" level=fatal msg="Get http:///var/run/docker.sock/v1.18/containers/json: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?" . ok ============================================================================ debian/changelog message for that: * Check if docker daemon runs before sending commands (Bug #38307) Otherwise seems to work.
(In reply to Daniel Tröder from comment #5) > (IMO this kind of belongs to #38307, but I fixed it referencing this bug.) > > I moved the Docker container rules to the DOCKER chain, so the Docker engine > removes them when it shuts down containers. > > Commit: 64626 see Bug #39429 The docker rules are still moved to the FORWARD chain by univention-firewall restart. Is that OK? -> /etc/init.d/docker start Chain FORWARD (policy ACCEPT) target prot opt source destination DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere Chain DOCKER (1 references) target prot opt source destination ACCEPT tcp -- anywhere 172.17.0.1 tcp dpt:https ACCEPT tcp -- anywhere 172.17.0.1 tcp dpt:http ->/etc/init.d/univention-firewall restart Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT tcp -- anywhere 172.17.0.1 tcp dpt:https ACCEPT tcp -- anywhere 172.17.0.1 tcp dpt:http Chain OUTPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere Chain DOCKER (0 references) target prot opt source destination
Hmm, and I think there is a second issue: When univention-firewall is stopped, the containers loose their connection: ============================================================= root@master60:~# docker exec -it fff8bf71a5b5 ping -c1 -W3 www.google.de PING www.google.de (173.194.112.31) 56(84) bytes of data. 64 bytes from fra07s27-in-f31.1e100.net (173.194.112.31): icmp_req=1 ttl=49 time=12.1 ms --- www.google.de ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 12.154/12.154/12.154/0.000 ms root@master60:~# /etc/init.d/univention-firewall stop Stopping Univention iptables configuration::. root@master60:~# docker exec -it fff8bf71a5b5 ping -c1 -W3 www.google.de PING www.google.de (173.194.112.23) 56(84) bytes of data. root@master60:~# =============================================================
(In reply to Arvid Requate from comment #8) > Hmm, and I think there is a second issue: When univention-firewall is > stopped, the containers loose their connection: For UCS 4.1-0 we will keep the current status. This issue is addressed later on in a separate bug. If "univention-firewall" is stopped, all containers will loose connectivity until "docker" is restarted or the system is rebooted (if univention-firewall is deactivated permanently via security/packetfilter/disabled=yes). (In reply to Felix Botner from comment #7) > The docker rules are still moved to the FORWARD chain by univention-firewall > restart. Is that OK? Fixed. Additionally the DOCKER queue is created if missing. univention-firewall (7.0.0-8): r65007 | Bug #38307: create DOCKER queue if missing / simplified code Successful build Package: univention-firewall Version: 7.0.0-8.84.201510301351 User: sschwardt Branch: ucs_4.1-0
OK: univention-firewall restart recreates the same rules are before OK: docker removes rules when it shuts down a container Tests: 1. boot with no docker containers, with docker and univention-firewall started 2. start a container (dudle-docker) # iptables -L -n -v Chain INPUT (policy DROP 0 packets, 0 bytes) [no rule changed] Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) [no rule changed] Chain OUTPUT (policy ACCEPT 289 packets, 70180 bytes) [no rule changed] Chain DOCKER (1 references) + 0 0 ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.1 tcp dpt:443 + 0 0 ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.1 tcp dpt:80 # iptables -L -n -v -t nat Chain POSTROUTING (policy ACCEPT 3 packets, 442 bytes) + 0 0 MASQUERADE tcp -- * * 172.17.0.1 172.17.0.1 tcp dpt:443 + 0 0 MASQUERADE tcp -- * * 172.17.0.1 172.17.0.1 tcp dpt:80 Chain DOCKER (2 references) + 0 0 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:40001 to:172.17.0.1:443 + 0 0 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:40000 to:172.17.0.1:80 3. restart univention-firewall # /etc/init.d/univention-firewall restart # iptables -L -n -v Chain INPUT (policy DROP 0 packets, 0 bytes) [no rule changed] Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) [no rule changed] Chain OUTPUT (policy ACCEPT 289 packets, 70180 bytes) [no rule changed] Chain DOCKER (1 references) [no rule changed] # iptables -L -n -v -t nat Chain POSTROUTING (policy ACCEPT 23 packets, 1380 bytes) [no rule changed] Chain DOCKER (2 references) [no rule changed] 4. stop and start container #/etc/init.d/docker-app-dudle-docker stop #/etc/init.d/docker-app-dudle-docker start # iptables -L -n -v Chain INPUT (policy DROP 0 packets, 0 bytes) [no rule changed] Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) [no rule changed] Chain OUTPUT (policy ACCEPT 289 packets, 70180 bytes) [no rule changed] Chain DOCKER (1 references) [no rule changed, only IP of container changed] # iptables -L -n -v -t nat Chain POSTROUTING (policy ACCEPT 23 packets, 1380 bytes) [no rule changed, only IP of container changed] Chain DOCKER (2 references) [no rule changed, only IP of container changed] 5. restart univention-firewall # /etc/init.d/univention-firewall restart # iptables -L -n -v Chain INPUT (policy DROP 0 packets, 0 bytes) [no rule changed] Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) [no rule changed] Chain OUTPUT (policy ACCEPT 289 packets, 70180 bytes) [no rule changed] Chain DOCKER (1 references) [no rule changed] # iptables -L -n -v -t nat Chain POSTROUTING (policy ACCEPT 23 packets, 1380 bytes) [no rule changed] Chain DOCKER (2 references) [no rule changed] 6. stop container → docker removes rules # /etc/init.d/docker-app-dudle-docker stop # iptables -L -n -v Chain INPUT (policy DROP 0 packets, 0 bytes) [no rule changed] Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) [no rule changed] Chain OUTPUT (policy ACCEPT 289 packets, 70180 bytes) [no rule changed] Chain DOCKER (1 references) - 0 0 ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.3 tcp dpt:443 - 0 0 ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.3 tcp dpt:80 # iptables -L -n -v -t nat Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) - 0 0 MASQUERADE tcp -- * * 172.17.0.3 172.17.0.3 tcp dpt:443 - 0 0 MASQUERADE tcp -- * * 172.17.0.3 172.17.0.3 tcp dpt:80 Chain DOCKER (2 references) - 0 0 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:40001 to:172.17.0.3:443 - 0 0 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:40000 to:172.17.0.3:80 ------------------------------------------------------------------- (In reply to Sönke Schwardt-Krummrich from comment #9) > (In reply to Arvid Requate from comment #8) > > Hmm, and I think there is a second issue: When univention-firewall is > > stopped, the containers loose their connection: > For UCS 4.1-0 we will keep the current status. This issue is addressed later > on in a separate bug. → Bug #39686
UCS 4.1 has been released: https://docs.software-univention.de/release-notes-4.1-0-en.html https://docs.software-univention.de/release-notes-4.1-0-de.html If this error occurs again, please use "Clone This Bug".
Gah... docker dev (although not many commits) dismisses the non-manageability of dockers iptables handling: https://github.com/docker/docker/issues/12791