Univention Bugzilla – Bug 51316
base/univention-updater: migrate to python3
Last modified: 2021-05-25 16:01:14 CEST
The UMC module of base/univention-updater has to be python3-compatible.
Part of git:phahn/51316-updater3
commit 2b1eea87b441038010393a8b95c851f23456bd04 (HEAD -> 5.0-0, origin/5.0-0) Merge: 6a1686c10b 4906566dd9 Author: Philipp Hahn <hahn@univention.de> Date: Mon Feb 1 18:15:36 2021 +0100 Bug #51316: Updater Python 3 Merge branch 'phahn/51316-updater3' into 5.0-0 Package: ucslint Version: 9.0.1-2A~5.0.0.202102011817 Package: univention-dvd Version: 5.0.1-1A~5.0.0.202102011821 Package: univention-lib Version: 9.0.11-1A~5.0.0.202102011824 Package: univention-management-console Version: 12.0.8-1A~5.0.0.202102011828 Package: ucs-test Version: 10.0.3-1A~5.0.0.202102011834 Package: univention-directory-listener Version: 14.0.5-1A~5.0.0.202102011843 Package: univention-updater Version: 15.0.3-1A~5.0.0.202102011847 Package: univention-appcenter Version: 9.0.2-1A~5.0.0.202102011852
5.0-0] 4906566dd9 Bug #51316 up: Fix debian package dependencies [5.0-0] 083c9fa5fd Bug #51316 up: Convert scripts to Python modules [5.0-0] 80595910c4 Bug #51316 up: Convert u-updater to Python 3 [5.0-0] a5ad528607 Bug #51316 up: Convert u-upgrade to Python 3 [5.0-0] 95f71afd10 Bug #51316 up: Convert statistics to Python3 [5.0-0] 057e510658 Bug #51316 up: Convert u-prune-kernels to Python 3 [5.0-0] 11cc44395c Bug #51316 up: Convert u-repo-update to Python3 [5.0-0] 2d6af834be Bug #51316 up: Convert u-repo-create to Python 3 [5.0-0] 3c81608488 Bug #51316 up: Convert UMC to Python 3 [5.0-0] bc40a19711 Bug #51316 up: Remove unused UMC commands [5.0-0] 5ba6636943 Bug #51316 up: Add docker-compose for dev [5.0-0] de60cb0f0f Bug #51316 up: Remove override_dh_fixperms [5.0-0] 64cd9e63ff Bug #51316 up: Fix temporary file handling [5.0-0] 544776636e Bug #51316 up: Merge into get_sh_files() [5.0-0] f400f2b29f Bug #51316 up: Unify logging [5.0-0] 5d65fa912c Bug #51316 up: Convert to pytest [5.0-0] e305aac250 Bug #51316 up: Remove UCRV repository/online/unmaintained [5.0-0] 34aae2d69e Bug #51316 up: Remove `.univention_install` file [5.0-0] a29e080c4c Bug #51316 up: Use python apt [5.0-0] 470d85cef2 Bug #51316 up: Remove apt-get upgrade -u [5.0-0] afd2c9b3f4 Bug #51316 up: Fix package dependencies [5.0-0] f1212f155f Bug #51316 up: Fix shellcheck errors [5.0-0] 49f88324fc Bug #51316 up: Fix JS errors [5.0-0] 7839171b97 Bug #51316 up: Remove _iterate_{release|versions}() [5.0-0] 8657bb9def Bug #51316 up: Print component repositories [5.0-0] 63c7d209a9 Bug #51316 up: Merge get_components() [5.0-0] e6924d5978 Bug #51316 up: Extract component code [5.0-0] d83f2c1be8 Bug #51316 up: Setup default releases.json for tests [5.0-0] 4b3dc5757d Bug #51316 up: Make test verbose by default [5.0-0] 382bddf2ed Bug #51316 up: Fix urllib password [5.0-0] ef638fc3cc Bug #51316 up: Drop arch and parts from iterator [5.0-0] 4d35372abe Bug #51316 up: Remove unused UCSRepoDist [5.0-0] 557efec94e Bug #51316 up: Drop extra version info [5.0-0] 19426b20e9 Bug #51316 up: Rewrite get_next_version() [5.0-0] 0e037a7453 Bug #51316 up: Cache releases.json [5.0-0] 702169693f Bug #51316 up: Fix subprocess(shell=false) [5.0-0] f279798dbc Bug #51316 up: Use bytes for transferred data [5.0-0] 79af0cd68d Bug #51316 up: Remove get_current_version() [5.0-0] 129e93dfc8 Bug #51316 up: Return UCS_Version instead of str [5.0-0] 4d9e3e6bad Bug #51316 up: Always print repositories [5.0-0] 624f202193 Bug #51316 up: Make current component default [5.0-0] b7b311ab06 Bug #51316 up: Remove unused get_component() [5.0-0] 0d1f9166dd Bug #51316 up: is_component_defaultpackage_installed() [5.0-0] a4755ace5d Bug #51316 up: Remove component_update_available() [5.0-0] 941a8be43a Bug #51316 up: Convert run_dist_upgrade() [5.0-0] 1119c2b258 Bug #51316 up: Convert component_update_get_packages() [5.0-0] 87f6e3d8a7 Bug #51316 up: Convert UserAgent to Python3 [5.0-0] d7c90c7e34 Bug #51316 up: Simple cleanups [5.0-0] ee5caaed2e Bug #51316 up: More repo-create [5.0-0] 8a2499ce01 Bug #51316 up: Remove check for components [5.0-0] 807d6ce7c0 Bug #51316 up: Remove apt.conf.d/99updatecheck.conf [5.0-0] 0127e2afce Bug #51316 up: Fix bash completion [5.0-0] 8b796d9039 Bug #51316 up: Convert UDL license_uuid to Python 3 [5.0-0] 7f52b1a89c Bug #51316 up: Convert server to policy [5.0-0] 6e27b4ea18 Bug #51316 up: Convert repository to policy [5.0-0] 536a824c22 Bug #51316 up: Convert maintenace to policy [5.0-0] ce76315fd2 Bug #51316 up: Convert actualise to Python3 [5.0-0] eb3688fb0a Bug #51316 up: Convert actualise to policy [5.0-0] 7c9f6200eb Bug #51316 up: Extract apt locking code [5.0-0] 8c2ab4d437 Bug #51316 up: Convert locking to Python 3 [5.0-0] 543052e535 Bug #51316 ucslint: Ignore more files [5.0-0] 148e8d525c Bug #51316 test: Fix linting [5.0-0] 2534adbf0b fix Bug #52210 test: Add `Raw` output format [5.0-0] 60beec79c0 fix Bug #42235 UMC: Remove broken upgrade code [5.0-0] d9a36cefe3 fix Bug #51588 up: Switch from gzip to xz [5.0-0] 72428a9f61 fix Bug #43686: Convert UDL from runit to systemd [5.0-0] 4c7113923c fix Bug #46600 lib: Fix PEP 484 type annotations [5.0-0] e3bc88b15c fix Bug #52512 lib: fix unittests/test_misc.py::test_groupname [5.0-0] 4115e20913 fix Bug #46600 lib: Name binary unit test cases [5.0-0] e833163b17 fix Bug #46600 lib: Fix UCS_Version comparison
https://jenkins.knut.univention.de:8181/job/UCS-5.0/job/UCS-5.0-0/view/Default/job/AutotestJoin/lastCompletedBuild/SambaVersion=s4,Systemrolle=master/testReport/03_ucr/37check-ucr-templates-py3-migration-status/master091/ failed: test_configfile_python_compatibility[/etc/univention/templates/files/etc/apt/mirror.list] test_configfile_python_compatibility[/etc/univention/templates/files/etc/apt/apt.conf.d/55user_agent] test_configfile_python_compatibility[/etc/univention/templates/files/etc/apt/sources.list.d/20_ucs-online-component.list] test_configfile_python_compatibility[/etc/univention/templates/files/etc/apt/sources.list.d/15_ucs-online-version.list] These UCR templates are either not Py2+Py3 compatible or are not idempotent. If they are expected to return different output on each `ucr commit` you can add them to the list in the top of the test case.
Probably related: https://jenkins.knut.univention.de:8181/job/UCS-5.0/job/UCS-5.0-0/view/Default/job/AutotestJoin/lastCompletedBuild/SambaVersion=no-samba,Systemrolle=master-part-II/testReport/20_appcenter/20_can_apps_be_installed/master090/ [2021-02-02 01:52:35.505433] # Component 5.0-0-errata-test: Error downloading http://updates-test.software-univention.de/5.0/maintained/component/5.0-0-errata-test/amd64/Packages.gz: 404 [2021-02-02 01:52:35.505459] deb https://appcenter.software-univention.de/univention-repository/5.0/maintained/component/ ucsschool_20200805135936/amd64/ [2021-02-02 01:52:35.505478] deb https://appcenter.software-univention.de/univention-repository/5.0/maintained/component/ ucsschool_20200805135936/all/ [2021-02-02 01:52:35.505517] [2021-02-02 01:52:35.505776] b'Paketlisten werden gelesen...\nAbh\xc3\xa4ngigkeitsbaum wird aufgebaut....\nStatusinformationen werden eingelesen....\nEinige Pakete konnten nicht installiert werden. Das kann bedeuten, dass\nSie eine unm\xc3\xb6gliche Situation angefordert haben oder, wenn Sie die\nUnstable-Distribution verwenden, dass einige erforderliche Pakete noch\nnicht erstellt wurden oder Incoming noch nicht verlassen haben.\nDie folgenden Informationen helfen Ihnen vielleicht, die Situation zu l\xc3\xb6sen:\n\nDie folgenden Pakete haben unerf\xc3\xbcllte Abh\xc3\xa4ngigkeiten:\n ucs-school-umc-installer : H\xc3\xa4ngt ab von: python-ucs-school soll aber nicht installiert werden\nE: Probleme k\xc3\xb6nnen nicht korrigiert werden, Sie haben zur\xc3\xbcckgehaltene defekte Pakete.\n' None
(In reply to Florian Best from comment #4) > https://jenkins.knut.univention.de:8181/job/UCS-5.0/job/UCS-5.0-0/view/ > Default/job/AutotestJoin/lastCompletedBuild/SambaVersion=s4, > Systemrolle=master/testReport/03_ucr/37check-ucr-templates-py3-migration- > status/master091/ failed: [5.0-0] 5c96323a37 chore[test]: Rewrite 03_ucr/37check-ucr-templates test/ucs-test/debian/changelog | 6 ++ .../tests/03_ucr/37check-ucr-templates-py3-migration-status.py | 158 +++++++++++++++++++--------- 2 files changed, 114 insertions(+), 50 deletions(-) Package: ucs-test Version: 10.0.3-9A~5.0.0.202102051802 Branch: ucs_5.0-0 (In reply to Florian Best from comment #5) > Probably related: > > https://jenkins.knut.univention.de:8181/job/UCS-5.0/job/UCS-5.0-0/view/ > Default/job/AutotestJoin/lastCompletedBuild/SambaVersion=no-samba, > Systemrolle=master-part-II/testReport/20_appcenter/20_can_apps_be_installed/ > master090/ No: U@S tries to install ucs-school-umc-installer → python-ucs-school → python-support, which is no longer available with UCS-5: xen1.knut.univention.de:8000/packages/source/python-support
> +IGNORE = { > + # python-univention-updater is Python 3 only > + '/etc/univention/templates/files/etc/apt/apt.conf.d/55user_agent': {2}, > + '/etc/univention/templates/files/etc/apt/mirror.list': {2}, > + '/etc/univention/templates/files/etc/apt/sources.list.d/15_ucs-online-version.list': {2}, > + '/etc/univention/templates/files/etc/apt/sources.list.d/20_ucs-online-component.list': {2}, > +} You can't ignore those files. UCR templates must retain Python 2 compatible as every process can execute UCR under Python 2 and recompile them until UCS 5.1 / 5.0-1.
(In reply to Florian Best from comment #7) > > +IGNORE = { > > + # python-univention-updater is Python 3 only ... > You can't ignore those files. UCR templates must retain Python 2 compatible > as every process can execute UCR under Python 2 and recompile them until UCS > 5.1 / 5.0-1. Dam. Fixed: [5.0-0] 1cee4aafba Bug #51316 up: Re-add dependency on Python 2 modules for UCS-5.0 base/univention-updater/debian/changelog | 6 ++++++ base/univention-updater/debian/control | 2 +- doc/changelog/changelog-5.0-0.xml | 32 ++++++++++++++--------------- doc/changelog/release-notes-5.0-0-de.xml | 2 +- test/ucs-test/debian/changelog | 6 ++++++ .../tests/03_ucr/37check-ucr-templates-py3-migration-status.py | 8 ++------ 6 files changed, 32 insertions(+), 24 deletions(-) Package: univention-updater Version: 15.0.3-3A~5.0.0.202102080805 Package: ucs-test Version: 10.0.3-11A~5.0.0.202102080823 OK: /usr/share/ucs-test/03_ucr/37check-ucr-templates-py3-migration-status.py -vf
<https://jenkins.knut.univention.de:8181/job/UCS-5.0/job/UCS-5.0-0/job/ADConnectorMultiEnv/Version=w2k19-english/ws/test/updater.log.1.gz> > Traceback (most recent call last): > File "/usr/lib/python3/dist-packages/univention/updater/scripts/upgrade.py", line 442, in do_update > update_available = performUpdate(options, checkForUpdates=options.check, silent=False) > File "/usr/lib/python3/dist-packages/univention/updater/scripts/upgrade.py", line 139, in performUpdate > if func(options, checkForUpdates, silent): > File "/usr/lib/python3/dist-packages/univention/updater/scripts/upgrade.py", line 152, in do_release_update > version_next = updater.release_update_available() > File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1413, in release_update_available > return self.get_next_version(UCS_Version(ucs_version), components, errorsto) > File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1350, in get_next_version > ver = min(ver for ver, _data in self.get_releases() if ver > version) > File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1350, in <genexpr> > ver = min(ver for ver, _data in self.get_releases() if ver > version) > File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1322, in get_releases > for major_release in self.releases.get('releases', []): > AttributeError: 'UniventionUpdater' object has no attribute 'releases' `updater.releases` is loaded from `releases.json` on instance creation, but is does not happen when UCRV `repository/online=no`: modules/univention/updater/scripts/upgrade.py 148 updater = UniventionUpdater() modules/univention/updater/tools.py 1206 def __init__(self, check_access=True): 1220 self.ucr_reinit() 1234 def ucr_reinit(self): 1264 if not self.online_repository: 1267 return 1278 self._get_releases() 1280 def _get_releases(self): 1300 self.releases = json.loads(data) 1309 self.releases = {"error": str(e)} 1312 self.releases = {"error": str(exc)} Reproducer: ucr set repository/online=no univention-upgrade --disable-app-updates tail -f /var/log/univention/updater.log [5.0-0] e7e236013f fix[up] Initialize releases when offline base/univention-updater/debian/changelog | 6 ++++++ base/univention-updater/modules/univention/updater/tools.py | 1 + base/univention-updater/tests/test_updater.py | 9 +++++++++ 3 files changed, 16 insertions(+) Package: univention-updater Version: 15.0.3-4A~5.0.0.202102110812 Branch: ucs_5.0-0
https://jenkins.knut.univention.de:8181/job/UCS-5.0/job/UCS-5.0-0/job/AutotestJoin/lastCompletedBuild/SambaVersion=s4,Systemrolle=master/testReport/03_ucr/37check-ucr-templates-py3-migration-status/master091/ The test case fails sometimes, because of different output during error handling: Python 2 seems to always print the "all" URL, while Python 3 switches between "all" and "amd64". -# Component 5.0-0-errata-test: Error downloading https://updates-test.software-univention.de/5.0/maintained/component/5.0-0-errata-test/all/Packages.gz: 404 +# Component 5.0-0-errata-test: Error downloading https://updates-test.software-univention.de/5.0/maintained/component/5.0-0-errata-test/amd64/Packages.gz: 404 Reproduce: ucr set repository/online/component/5.0-0-errata-test=enabled repository/online/component/5.0-0-errata-test/unmaintained=disabled python3 $(which ucr) filter < /etc/univention/templates/files/etc/apt/sources.list.d/20_ucs-online-component.list If it's not worth to fix it, please add it to the ignore-list of the test.
[5.0-0] 62ce9d387c Bug #51316 up: Generate stable output .../conffiles/etc/apt/sources.list.d/20_ucs-online-component.list | 2 +- base/univention-updater/debian/changelog | 6 ++++++ base/univention-updater/modules/univention/updater/tools.py | 25 +++++++++++++++++++++++-- base/univention-updater/tests/test_updater.py | 25 +++++++++++++++++++++++++ 4 files changed, 55 insertions(+), 3 deletions(-) Package: univention-updater Version: 15.0.3-5A~5.0.0.202102151341 Branch: ucs_5.0-0 OK: ucs-test -E dangerous -F raw -s updater OK: /usr/share/ucs-test/03_ucr/37check-ucr-templates-py3-migration-status.py -vf
Changes -> OK flake8 -> OK umc upgrade -> OK univention-upgrade -> OK app installation -> OK Tests -> OK Changelog -> OK
UCS 5.0 has been released: https://docs.software-univention.de/release-notes-5.0-0-en.html https://docs.software-univention.de/release-notes-5.0-0-de.html If this error occurs again, please use "Clone This Bug".