Bug 30590 - Package upgrade removes interfaces of running VMs from bridge
Package upgrade removes interfaces of running VMs from bridge
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: Virtualization - KVM
UCS 3.1
All Linux
: P3 normal (vote)
: UCS 3.1-0-errata
Assigned To: Philipp Hahn
Stefan Gohmann
:
Depends on:
Blocks: 39334
  Show dependency treegraph
 
Reported: 2013-02-26 10:22 CET by Philipp Hahn
Modified: 2017-04-21 16:29 CEST (History)
2 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional): Troubleshooting
Max CVSS v3 score:
hahn: Patch_Available+


Attachments
Do not restart bridge (1.39 KB, patch)
2013-02-26 10:22 CET, Philipp Hahn
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hahn univentionstaff 2013-02-26 10:22:56 CET
Created attachment 5096 [details]
Do not restart bridge

An upgrade of univention-virtual-machine-manager-node-kvm stops and restarts
the bridge, which forcefully removes all the tap-interfaces of all running qemu
processes from the bridge. On restart the state is not restored, so all VMs are
no longer connected to the network.

virsh # domiflist janis_OpenERP-Master
vnet0      bridge     eth0       virtio
virsh # domiflist janis_OpenERP-Client
vnet1      bridge     eth0       virtio
# brctl show
bridge name     bridge id               STP enabled     interfaces
eth0            8000.0019997390a8       no              peth0
                                                        vnet3

Manual fix:
# brctl addif eth0 vnet0
# brctl addif eth0 vnet1
Comment 1 Philipp Hahn univentionstaff 2013-02-26 11:26:21 CET
3.1-1: svn39246, univention-virtual-machine-manager-node_2.0.3-2.63.201302261121
ChangeLog: svn16580
\item On upgrades of \ucsName{univention-virtual-machine-manager-node-kvm} the default network bridge for virtual machines was shutdown, which disconnects the interfaces of all runnings VMs. This is no longer done on upgrades (\ucsBug{30590}).

3.1-0e: svn11523, univention-virtual-machine-manager-node_2.0.2-2.62.201302261118
2013-02-26-univention-virtual-machine-manager-node.yaml svn16580

# demesg
[6345455.049321] eth0: port 2(vnet0) entering forwarding state
[6345455.084105] eth0: port 4(vnet2) entering forwarding state
[6345455.118092] eth0: port 6(vnet4) entering forwarding state
[6345455.151352] eth0: port 3(vnet1) entering forwarding state
[6345455.184650] eth0: port 1(peth0) entering forwarding state
[6345455.439207] device peth0 left promiscuous mode
[6345455.471480] eth0: port 1(peth0) entering disabled state
[6345455.868645] ADDRCONF(NETDEV_UP): eth0: link is not ready
[6345458.676259] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[6345458.711283] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[6345469.046549] eth0: no IPv6 routers present
[6345494.062916] device vnet0 left promiscuous mode
[6345494.097108] tmpbridge: port 2(vnet0) entering disabled state
[6345494.132377] device vnet2 left promiscuous mode
[6345494.166772] tmpbridge: port 4(vnet2) entering disabled state
[6345494.203048] device vnet4 left promiscuous mode
[6345494.238274] tmpbridge: port 6(vnet4) entering disabled state
[6345494.275465] device vnet1 left promiscuous mode
[6345494.311674] tmpbridge: port 3(vnet1) entering disabled state
[6345523.965143] ADDRCONF(NETDEV_UP): peth0: link is not ready
[6345526.343330] igb: peth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[6345526.383318] ADDRCONF(NETDEV_CHANGE): peth0: link becomes ready
[6345527.018299] device peth0 entered promiscuous mode
[6345527.235061] eth0: port 1(peth0) entering forwarding state
[6345527.274104] eth0: port 1(peth0) entering forwarding state
[6345536.861191] peth0: no IPv6 routers present
[6345538.274894] eth0: no IPv6 routers present

Similar issue at krus:

# demesg
[6346981.962239] eth0: port 32(vnet30) entering forwarding state
[6346982.002428] eth0: port 31(vnet29) entering forwarding state
[6346982.041936] eth0: port 30(vnet28) entering forwarding state
[6346982.080409] eth0: port 26(vnet24) entering forwarding state
[6346982.118496] eth0: port 25(vnet23) entering forwarding state
[6346982.156034] eth0: port 21(vnet19) entering forwarding state
[6346982.192939] eth0: port 14(vnet12) entering forwarding state
[6346982.228883] eth0: port 6(vnet4) entering forwarding state
[6346982.263908] eth0: port 4(vnet2) entering forwarding state
[6346982.297859] eth0: port 10(vnet8) entering forwarding state
[6346982.332316] eth0: port 5(vnet3) entering forwarding state
[6346982.366569] eth0: port 1(peth0) entering forwarding state
[6346982.514492] device peth0 left promiscuous mode
[6346982.547392] eth0: port 1(peth0) entering disabled state
[6346982.986474] ADDRCONF(NETDEV_UP): eth0: link is not ready
[6346986.461300] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[6346986.498802] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[6347020.649295] device vnet30 left promiscuous mode
[6347020.683955] tmpbridge: port 32(vnet30) entering disabled state
[6347020.719821] device vnet29 left promiscuous mode
[6347020.755281] tmpbridge: port 31(vnet29) entering disabled state
[6347020.791440] device vnet28 left promiscuous mode
[6347020.827856] tmpbridge: port 30(vnet28) entering disabled state
[6347020.864952] device vnet24 left promiscuous mode
[6347020.902107] tmpbridge: port 26(vnet24) entering disabled state
[6347020.940424] device vnet23 left promiscuous mode
[6347020.978432] tmpbridge: port 25(vnet23) entering disabled state
[6347021.017119] device vnet19 left promiscuous mode
[6347021.057153] tmpbridge: port 21(vnet19) entering disabled state
[6347021.097730] device vnet12 left promiscuous mode
[6347021.137925] tmpbridge: port 14(vnet12) entering disabled state
[6347021.179363] device vnet4 left promiscuous mode
[6347021.220524] tmpbridge: port 6(vnet4) entering disabled state
[6347021.262314] device vnet2 left promiscuous mode
[6347021.303210] tmpbridge: port 4(vnet2) entering disabled state
[6347021.344750] device vnet8 left promiscuous mode
[6347021.387011] tmpbridge: port 10(vnet8) entering disabled state
[6347021.430354] device vnet3 left promiscuous mode
[6347021.474401] tmpbridge: port 5(vnet3) entering disabled state
[6347039.299917] ADDRCONF(NETDEV_UP): peth0: link is not ready
[6347042.919925] igb: peth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[6347042.970275] ADDRCONF(NETDEV_CHANGE): peth0: link becomes ready
[6347043.367786] device peth0 entered promiscuous mode
[6347043.612682] eth0: port 1(peth0) entering forwarding state
[6347043.662526] eth0: port 1(peth0) entering forwarding state
[6347053.837395] eth0: no IPv6 routers present
Comment 2 Philipp Hahn univentionstaff 2013-02-26 11:59:21 CET
The patch is insufficient: the already-installed prerm is broken, there is no
clean way to fix that.
Comment 3 Philipp Hahn univentionstaff 2013-02-26 16:51:04 CET
uvmm-node-kvm now contains a script /usr/lib/univention-virtual-machine-manager-node-kvm/ucs-kvm-reconnect-bridge, which is called from postinst on upgrades from <= 2.0.2-2.61.201301252219, which is the current broken version in errata3.1-0.
It restores all interfaces to the bridges as libvirt known (this might overwrite any manual modification done by brctl, which is unsupported.)

3.1-1: svn39262
errata3.1-0: svn11529, 2.0.2-2.64.201302261642
2013-02-26-univention-virtual-machine-manager-node.yaml: svn16586

For QA the script on its own:
# virsh domiflist test
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      bridge     br0        virtio      52:54:00:a5:4c:14
# virsh domiflist test2
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet1      bridge     br0        virtio      52:54:00:88:14:47
vnet2      bridge     br0        virtio      52:54:00:97:ec:1a
vnet3      bridge     br0        virtio      52:54:00:fc:ab:f2
# brctrl show 
br0             8000.bcaec507cc5c       no              eth0
                                                        vnet0
                                                        vnet1
                                                        vnet2
                                                        vnet3
# brctl delif br0 vnet0
# brctl delif br0 vnet2
# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.bcaec507cc5c       no              eth0
                                                        vnet1
                                                        vnet3
# ./ucs-kvm-reconnect-bridge
# brctrl show 
br0             8000.bcaec507cc5c       no              eth0
                                                        vnet0
                                                        vnet1
                                                        vnet2
                                                        vnet3
Comment 4 Stefan Gohmann univentionstaff 2013-02-27 07:07:15 CET
From the logfile during the errata update:

/var/lib/dpkg/info/univention-virtual-machine-manager-node-kvm.postinst: 60: /usr/lib/univention-virtual-machine-manager-node-kvm/ucs-kvm-reconnect-bridge: Permission denied

The network connection to the VMs was already in the preinst stopped. Maybe it would be better to call the script in the preinst already?
Comment 5 Philipp Hahn univentionstaff 2013-02-27 09:14:19 CET
(In reply to comment #4)
> From the logfile during the errata update:
> 
> /var/lib/dpkg/info/univention-virtual-machine-manager-node-kvm.postinst: 60:
> /usr/lib/univention-virtual-machine-manager-node-kvm/ucs-kvm-reconnect-bridge:
> Permission denied

This is a problem causes by the patch system: The script in branch-3.1 is executable due the use of svn:executable, which is not representable in the patch file format. As a work-around the script is explicitly prefix by invocating python2.6

> The network connection to the VMs was already in the preinst stopped. Maybe it
> would be better to call the script in the preinst already?

The script is not yet unpacked and thus not available in preinst.

errata3.1-0: svn11533, 2.0.2-2.66.201302270909
2013-02-26-univention-virtual-machine-manager-node.yaml svn16593
Comment 6 Philipp Hahn univentionstaff 2013-02-27 10:17:20 CET
The script must only be called after the init script has been called, since it is responsible for creating the bridge.

3.1-1: svn39270, TBB
e3.1-0: svn11534, 2.0.2-2.67.201302270956
 2013-02-26-univention-virtual-machine-manager-node.yaml svn16594
Comment 7 Stefan Gohmann univentionstaff 2013-02-27 11:19:33 CET
Tests: OK

YAML + Changelog: OK
Comment 8 Moritz Muehlenhoff univentionstaff 2013-02-27 12:12:42 CET
http://errata.univention.de/3.1-errata57.html