Bug 45998 - update pyopenssl and python-cryptography version in ucs-school-4.2 scope to match requirement of twisted
update pyopenssl and python-cryptography version in ucs-school-4.2 scope to m...
Status: CLOSED FIXED
Product: UCS@school
Classification: Unclassified
Component: General
UCS@school 4.2
Other Linux
: P5 normal (vote)
: UCS@school 4.2 v9
Assigned To: Daniel Tröder
Sönke Schwardt-Krummrich
:
Depends on:
Blocks: 46024
  Show dependency treegraph
 
Reported: 2018-01-10 13:15 CET by Daniel Tröder
Modified: 2018-05-02 18:18 CEST (History)
2 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 3: Simply Wrong: The implementation doesn't match the docu
Who will be affected by this bug?: 2: Will only affect a few installed domains
How will those affected feel about the bug?: 4: A User would return the product
User Pain: 0.137
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Tröder univentionstaff 2018-01-10 13:15:27 CET
The version of pyopenssl and python-cryptography in the ucs-school-4.2 scope does not match the expected version of twisted v16, but of twisted v14.

This creates a traceback in the OPSI app.

Update pyopenssl and python-cryptography.
Comment 1 Florian Best univentionstaff 2018-01-10 13:34:06 CET
Can you attach the traceback for searchability?
Comment 2 Daniel Tröder univentionstaff 2018-01-10 13:37:12 CET
From https://help.univention.com/t/ucs-update-broke-opsi-app-python-cryptography-issue/7652

Jan 6 09:03:58 hostname opsiconfd[5904]: Traceback (most recent call last):
Jan 6 09:03:58 hostname opsiconfd[5904]: File “/usr/bin/opsiconfd”, line 12, in 
Jan 6 09:03:58 hostname opsiconfd[5904]: from opsiconfd.opsiconfd import main
Jan 6 09:03:58 hostname opsiconfd[5904]: File “/usr/lib/python2.7/dist-packages/opsiconfd/opsiconfd.py”, line 68, in 
Jan 6 09:03:58 hostname opsiconfd[5904]: from OPSI.Backend.BackendManager import BackendManager
Jan 6 09:03:58 hostname opsiconfd[5904]: File “/usr/lib/python2.7/dist-packages/OPSI/Backend/BackendManager.py”, line 40, in 
Jan 6 09:03:58 hostname opsiconfd[5904]: from OPSI.Backend.Backend import (Backend, BackendModificationListener,
Jan 6 09:03:58 hostname opsiconfd[5904]: File “/usr/lib/python2.7/dist-packages/OPSI/Backend/Backend.py”, line 47, in 
Jan 6 09:03:58 hostname opsiconfd[5904]: from twisted.conch.ssh import keys
Jan 6 09:03:58 hostname opsiconfd[5904]: File “/usr/lib/python2.7/dist-packages/twisted/conch/ssh/keys.py”, line 25, in 
Jan 6 09:03:58 hostname opsiconfd[5904]: from cryptography.hazmat.primitives.asymmetric.utils import (
Jan 6 09:03:58 hostname opsiconfd[5904]: ImportError: No module named utils
Comment 3 Daniel Tröder univentionstaff 2018-01-11 09:37:27 CET
All required source packages were imported from jessie-BP to the ucs-school-4.2 scope.

As python-setuptools can currently not be build in our buildsystem (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879865), I copied it from 4.3-0. It is needed for building only.
debhelper>=10 and dpkg>=1.18 can also currently not be build in our buildsystem, so as it is needed for building only, it was also copied. And xz-tools and tar...

All these packages must *not* be copied to the appcenter repository!

dtroeder@dimma:/var/univention/buildsystem2/apt$ cp ucs_4.3-0/all/*setuptools*_33.1.1-1_all.deb ucs_4.3-0/all/*pkg-resources*_33.1.1-1_all.deb ucs_4.3-0/all/debhelper_10.2.5_all.deb ucs_4.2-0-ucs-school-4.2/all
dtroeder@dimma:/var/univention/buildsystem2/apt$ cp ucs_4.3-0/all/dpkg-dev_1.18.24_all.deb ucs_4.3-0/all/libdpkg-perl_1.18.24_all.deb ucs_4.2-0-ucs-school-4.2/all
dtroeder@dimma:/var/univention/buildsystem2/apt$ cp ucs_4.3-0/amd64/libdpkg-dev_1.18.24_amd64.deb ucs_4.2-0-ucs-school-4.2/amd64/
dtroeder@dimma:/var/univention/buildsystem2/apt$ cp ucs_4.3-0/amd64/dpkg_1.18.24_amd64.deb ucs_4.3-0/amd64/tar_1.29b-1.1_amd64.deb ucs_4.2-0-ucs-school-4.2/amd64/
dtroeder@dimma:/var/univention/buildsystem2/apt$ cp ucs_4.3-0/i386/dpkg_1.18.24_i386.deb ucs_4.3-0/i386/tar_1.29b-1.1_i386.deb ucs_4.2-0-ucs-school-4.2/i386/
dtroeder@dimma:/var/univention/buildsystem2/apt$ cp ucs_4.3-0/amd64/liblzma5_5.2.2-1.2+b1_amd64.deb ucs_4.3-0/amd64/liblzma-dev_5.2.2-1.2+b1_amd64.deb ucs_4.2-0-ucs-school-4.2/amd64/
dtroeder@dimma:/var/univention/buildsystem2/apt$ cp ucs_4.3-0/i386/liblzma5_5.2.2-1.2+b1_i386.deb ucs_4.3-0/i386/liblzma-dev_5.2.2-1.2+b1_i386.deb ucs_4.2-0-ucs-school-4.2/i386/

dtroeder@dimma:/var/univention/buildsystem2/apt$ repo-apt-ftparchive --release ucs_4.2-0-ucs-school-4.2

------

Then python-cryptography and its dependencies were build.

Branch: ucs_4.2-0
Scope: ucs-school-4.2

six 1.10.0-3~bpo8+1A~4.2.0.201801091453
python-idna 2.0-3~bpo8+1A~4.2.0.201801101727
python-ipaddress 1.0.16-1~bpo8+1A~4.2.0.201801091458
strip-nondeterminism 0.034-1~bpo8+1A~4.2.0.201801091541
dh-autoreconf 12~bpo8+1A~4.2.0.201801091544
pycparser 2.17-2~bpo8+1A~4.2.0.201801091837
pyasn1 0.1.9-1~bpo8+1A~4.2.0.201801101729
dh-python 2.20170125~bpo8+1A~4.2.0.201801091843
python-cffi 1.9.1-2~bpo8+1A~4.2.0.201801091845
python-cryptography-vectors 1.7.1-1~bpo8+1A~4.2.0.201801101648
python-py 1.4.31-1~bpo8+1A~4.2.0.201801101707
python-hypothesis 3.4.2-2~bpo8+1A~4.2.0.201801101710
pytest 3.0.3-1~bpo8+1A~4.2.0.201801101712
python-cryptography 1.7.1-3~bpo8+1A~4.2.0.201801101730
pyopenssl 16.0.0-1~bpo8+1A~4.2.0.201801101752


With v16 code was moved from python-twisted-conch to python-twisted-core. Runtime requirements of python-twisted-conch & -core are fulfilled now with python-openssl v16.0 and python-cryptography v1.7, both from jessie-backports (and their respective dependencies).

Test:
-----
# python -c 'import cryptography.hazmat.primitives.asymmetric.utils' && echo OK
OK
# univention-app install opsi
[..]
# # systemctl status opsiconfd.service 
● opsiconfd.service - Opsi Configuration Service
   Loaded: loaded (/lib/systemd/system/opsiconfd.service; enabled)
   Active: active (running) since Do 2018-01-11 09:02:16 CET; 26min ago
 Main PID: 10097 (opsiconfd)
   CGroup: /system.slice/opsiconfd.service
           └─10097 /usr/bin/python /usr/bin/opsiconfd -D
# ps xa | grep opsiconfd
 8945 ?        S      0:00 /bin/bash +e /usr/bin/opsiconfd-guard daemon
10097 ?        S      0:00 /usr/bin/python /usr/bin/opsiconfd -D
Comment 4 Daniel Tröder univentionstaff 2018-01-12 08:41:28 CET
Today I was greeted with this traceback, when I opened the UMC:

-------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/sbin/univention-management-console-web-server", line 235, in check_queue
    cls.dispatch(queuerequest)
  File "/usr/sbin/univention-management-console-web-server", line 250, in dispatch
    client = SessionClient(ip=queuerequest.ip)
  File "/usr/sbin/univention-management-console-web-server", line 126, in __init__
    self.client = Client()
  File "/usr/lib/pymodules/python2.7/univention/management/console/protocol/client.py", line 101, in __init__
    self.__crypto_context = SSL.Context(SSL.TLSv1_METHOD)
  File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 272, in __init__
    self._problems.append(e)
  File "/usr/lib/python2.7/dist-packages/cffi/api.py", line 312, in gc
ImportError: No module named gc_weakref
-------------------------------------------------------------------

I could reproduce it on a 2nd server, after updating python-cffi.

It went away after "service univention-management-console-web-server restart", as the univention-management-console-web-server was then using the new pyopenssl binaries.

I suggest to add the "umc-web-server restart" to ucs-school-lib/debian/python-ucs-school.postinst (with a conditional for updating to the current version).
Comment 5 Daniel Tröder univentionstaff 2018-01-12 12:01:57 CET
The binary packages that would be upgraded on a users system are:

python-openssl
python-cryptography
python-cffi
python-pyasn1
python-six
Comment 6 Daniel Tröder univentionstaff 2018-01-12 12:04:58 CET
python-enum34, python-ipaddress and python-idna do not exist in jessie.
python-pyasn1 would be upgraded from 0.1.7-1 to 0.1.9-1.

The most critical change happens in python-cryptography 0.8, where a lot of classes/functions are moved between Python modules (see https://cryptography.io/en/latest/changelog/#v0-8).
Comment 7 Florian Best univentionstaff 2018-01-12 12:17:24 CET
(In reply to Daniel Tröder from comment #5)
> The binary packages that would be upgraded on a users system are:
> 
> python-openssl
> python-cryptography
> python-cffi
> python-pyasn1
> python-six

Where do you want to upgrade these packages? in UCS? in UCS@school?
I think this might be problematic, we had even in the past multiple problems with python-psutil. If OPSI depends on a more recent Twisted version it should either use virtualenv or docker.
Comment 8 Niko Wenselowski 2018-01-12 12:38:52 CET
(In reply to Florian Best from comment #7)
> (In reply to Daniel Tröder from comment #5)
> > The binary packages that would be upgraded on a users system are:
> > 
> > python-openssl
> > python-cryptography
> > python-cffi
> > python-pyasn1
> > python-six
> 
> Where do you want to upgrade these packages? in UCS? in UCS@school?
> I think this might be problematic, we had even in the past multiple problems
> with python-psutil. If OPSI depends on a more recent Twisted version it
> should either use virtualenv or docker.
opsi runs with Twisted versions ranging from ancient 10.something until recent 17.x and works fine with the 14.x provided through UCS.
This is until someone installs UCS@school and updates his packages because the updated python-twisted-conch 16.x provided through the UCS@school repo does not work with python-cryptography provided through the core UCS repos.

I fail to see why opsi should make changes if the break is introduced through UCS@school.
Comment 9 Florian Best univentionstaff 2018-01-12 12:51:37 CET
(In reply to Niko Wenselowski from comment #8)
> opsi runs with Twisted versions ranging from ancient 10.something until
> recent 17.x and works fine with the 14.x provided through UCS.
> This is until someone installs UCS@school and updates his packages because
> the updated python-twisted-conch 16.x provided through the UCS@school repo
> does not work with python-cryptography provided through the core UCS repos.
> 
> I fail to see why opsi should make changes if the break is introduced
> through UCS@school.

Oh yes!
Then I think it might be good to consider if our UCS@school import could run in a virtualenv / docker environment instead of changing distribution packages.
Comment 10 Daniel Tröder univentionstaff 2018-01-12 13:42:15 CET
This has nothing to do with the import. UCS@school does not use twisted.
It was built as a build dependency and uploaded to the app center by accident. We are currently in the process of evaluating the best way how to deal with it: either downgrade twisted or upgrade python-openssl and python-cryptography.
Comment 11 Sönke Schwardt-Krummrich univentionstaff 2018-01-12 17:16:19 CET
As discussed:
the newer version of python-twisted is removed from UCS@school 4.2 v7 and therefore the update of python-cryptography et al. is not required anymore.

A new SDB article describes what has to be done to downgrade python-twisted:
https://help.univention.com/t/resolving-opsi-installation-update-issues-on-ucs-school-systems/7705

@Daniel: Please check if there are changes that have to be reverted.
Comment 12 Daniel Tröder univentionstaff 2018-01-15 16:04:34 CET
No commits were made to SVN/GIT.

I removed the following source packages from the buildsystems scope:

* enum34
* twisted
* python-cryptography
* pyopenssl
* python-ipaddress
* python-cffi
* pyasn1
* python-hypothesis
* strip-nondeterminism
* dh-autoreconf
* pycparser
* dh-python
* python-py

I removed the following binary packages from the repo:

* all/python-enum34*
* */*twisted*
* */python*-cryptography*.deb
* all/python*-openssl*.deb
* all/python-ipaddress_*.deb
* */python*-cffi*.deb
* all/python*-pyasn1*.deb
* all/python*-hypothesis*.deb
* all/*nondeterminism*.deb
* all/dh-autoreconf_*.deb
* all/python*-pycparser_*.deb
* all/dh-python_*.deb
* all/*python-py_1.4*.deb
---
* all/*pkg-resources*.deb
* all/debhelper_10.2.5_all.deb
* rm */*dpkg*_1.18.24*.deb
* rm */liblzma5_5.2.2-1.2+b1*.deb
Comment 13 Sönke Schwardt-Krummrich univentionstaff 2018-02-07 13:03:29 CET
OK: UCS@school 4.2 appcenter repo
OK: UCS@school 4.3 appcenter repo
??: scope ucs-school-4.2
??: scope ucs-school-4.3
OK: git/svn
Comment 14 Sönke Schwardt-Krummrich univentionstaff 2018-03-23 18:41:00 CET
(In reply to Sönke Schwardt-Krummrich from comment #13)
> ??: scope ucs-school-4.2
> ??: scope ucs-school-4.3
OK: scope ucs-school-4.2
OK: scope ucs-school-4.3