Univention Bugzilla – Bug 41840
Set update/available when release updates are available
Last modified: 2016-11-28 14:57:23 CET
The appbox image upgrades the container step wise. If univention-upgrade says that 4.1-1 is available, it upgrades to 4.1-1: Starting update process, this may take a while. [...] Checking for release updates: 4.1-2 is available but updater has been instructed to stop at version 4.1-1. Checking for package updates: found The following packages will be upgraded: [...] Starting package upgrade done Checking for release updates: 4.1-2 is available but updater has been instructed to stop at version 4.1-1. [...] Setting update/available This last line sets update/available=no for some reason. This is a problem because now "univention-app upgrade" stops because this very UCRV is checked before performing an expensive univention-upgrade --check. This needs to be fixed, otherwise one has to upgrade a container (potentially) multiple times, each time with a 1h delay (or so) until update/available is set back to the correct value (yes). When fixing this, please have a look at Bug#29642. This is the other way around, but in general update/available needs to be correctly set for Docker.
Fixed this by adjusting the return values of the three functions do_package_updates, do_app_updates, do_release_update Removed the last do_exec(), otherwise I could end up in an infinite loop. Intended to preserve the old functionality of re-executing itself after package/release update. After playing with it, I think --setucr should be the default, but this has not been touched. Bug#29642 is very much UMC based and may not be fixable in univention-upgrade itself. univention-updater 11.0.11-3.1483.201608160000
A lot of updater tests now fail, see for example 09_updater/23major.test: http://jenkins.knut.univention.de:8080/job/UCS-4.1/job/UCS-4.1-3/job/AutotestJoin/SambaVersion=s3,Systemrolle=master/19/testReport/09_updater/23major/test/ Please check your changes or adjust the test cases.
From the updater.log: Traceback (most recent call last): File "/usr/sbin/univention-upgrade", line 447, in do_update update_available = performUpdate(options, checkForUpdates=options.check, silent=False) File "/usr/sbin/univention-upgrade", line 150, in performUpdate update_available_local = func(options, checkForUpdates, silent) File "/usr/sbin/univention-upgrade", line 281, in do_app_updates app_upgrade_search.call_safe() AttributeError: type object 'UpgradeSearch' has no attribute 'call_safe' It looks like a dependency between the updater and the app center packages. That is at lest not defined in the YAML files and if possible it should be avoided.
Unfortunately, it is not possible to depend on a specific App Center package version in univention-updater. Instead, univention-appcenter depends on univention-updater (classes for handling Components, only legacy code IIRC). I raise an ImportError() if the univention-appcenter is not present at all. I have adjusted it so that this is raised also when call_safe() is not available. In this case univention-upgrade cannot upgrade Apps.
Hmmm... not so sure if this is the fix for the problem. The jenkins logs and my logs differ when I run 09_updater/23major locally. I guess somewhere a UCS 5.0 is mocked up.
(In reply to Dirk Wiesenthal from comment #5) > Hmmm... not so sure if this is the fix for the problem. The jenkins logs and > my logs differ when I run 09_updater/23major locally. I guess somewhere a > UCS 5.0 is mocked up. It is not only the 23major test case. Seven test cases on all roles fail now: 09_updater.20patch-single.test 09_updater.21patch-multiple.test 09_updater.22minor.test 09_updater.23major.test 09_updater.27updateto.test 09_updater.34updater-check.test 09_updater.49component-version-current-dep.test For me it is reproducible: ------------------------------------------------------------------------------ root@master411:~# /usr/share/ucs-test/09_updater/23major -f 6769 blocks gpg: keyring `/root/.gnupg/secring.gpg' created gpg: keyring `/root/.gnupg/pubring.gpg' created OK Stopping periodic command scheduler: cron. dpkg-preconfigure: kann Stdin nicht wieder öffnen: Datei oder Verzeichnis nicht gefunden Vormals nicht ausgewähltes Paket test-27201-26254 wird gewählt. (Lese Datenbank ... 101755 Dateien und Verzeichnisse sind derzeit installiert.) Entpacken von test-27201-26254 (aus .../test-27201-26254_1_amd64.deb) ... test-27201-26254 (1) wird eingerichtet ... Checking network repository Update to = 4.1-1 Checking network repository Update to = 4.2-0 Checking network repository Update to = 5.0-0 Checking network repository System is up to date (UCS 5.0-0) === RESULT: 100 === Starting 1 ucs-test at 2015-11-18 13:23:18 to /dev/null Update localhost repository one major version............................................................................................................... Test passed root@master411:~# ./activate-errata-test-scope.sh Create repository/online/component/4.1-3-errata-test/description Create repository/online/component/4.1-3-errata-test/version Create repository/online/component/4.1-3-errata-test/server Create repository/online/component/4.1-3-errata-test/username Create repository/online/component/4.1-3-errata-test/password Create repository/online/component/4.1-3-errata-test File: /etc/apt/sources.list.d/20_ucs-online-component.list File: /etc/apt/mirror.list File: /etc/apt/sources.list.d/15_ucs-online-version.list root@master411:~# univention-upgrade Starting univention-upgrade. Current UCS version is 4.1-3 errata234 Checking for local repository: none Checking for package updates: found The following packages will be installed: python-mysqldb The following packages will be upgraded: univention-directory-manager-tools,python-univention-directory-manager,python-univention-directory-manager-cli,python-univention-appcenter,univention-s4-connector,python-univention-connector-s4,univention-appcenter,univention-appcenter-docker,univention-management-console-module-updater,univention-updater,univention-management-console-module-appcenter,univention-management-console-module-apps Do you want to continue [Y|n]? y Starting package upgrade done Starting univention-upgrade. Current UCS version is 4.1-3 errata234 Checking for local repository: none Checking for package updates: none Checking for app updates: none Checking for release updates: none root@master411:~# /usr/share/ucs-test/09_updater/23major -f 6769 blocks gpg: keyring `/root/.gnupg/secring.gpg' created gpg: keyring `/root/.gnupg/pubring.gpg' created OK Stopping periodic command scheduler: cron. dpkg-preconfigure: kann Stdin nicht wieder öffnen: Datei oder Verzeichnis nicht gefunden Vormals nicht ausgewähltes Paket test-29434-24821 wird gewählt. (Lese Datenbank ... 101836 Dateien und Verzeichnisse sind derzeit installiert.) Entpacken von test-29434-24821 (aus .../test-29434-24821_1_amd64.deb) ... test-29434-24821 (1) wird eingerichtet ... Checking network repository Update to = 4.1-1 Checking network repository Update to = 4.2-0 Checking network repository Update to = 5.0-0 Checking network repository System is up to date (UCS 5.0-0) **************** Test failed above this line **************** ERROR 23major:50 0 ERROR test -n "$(ucr get update/available)" === RESULT: 110 === Starting 1 ucs-test at 2015-11-18 13:25:21 to /dev/null Update localhost repository one major version............................................................................................................... Test failed root@master411:~# dpkg -l | grep univention-updater ii univention-updater 11.0.11-4.1484.201608172305 all UCS - system upgrading tool root@master411:~# ------------------------------------------------------------------------------ (In reply to Dirk Wiesenthal from comment #4) > Unfortunately, it is not possible to depend on a specific App Center package > version in univention-updater. Instead, univention-appcenter depends on > univention-updater (classes for handling Components, only legacy code IIRC). > > I raise an ImportError() if the univention-appcenter is not present at all. > I have adjusted it so that this is raised also when call_safe() is not > available. In this case univention-upgrade cannot upgrade Apps. That means if we release the univention-updater package no one can install app updates via univention-upgrade until we release the appcenter package. That is for me a blocker. Why can't we check on code level if the function is available and use the old one if it is not available?
Okay, using old code, not depending on univention-appcenter update anymore. Got all the tests running (yesterday, I had a repo registered which caused the tests to fail before the interesting part). Fixed in univention-updater 11.0.11-5.1485.201608190011
It looks like a regression was introduced by these changes. The AD member mode Jenkins jobs uses updates-test.software-univention.de. Since a few days UCS 4.2 is available on updates-test. Anyway, the test case 00_checks.99check_log_files found the following entries: ****************************************************************************** *** BEGIN *** ['/usr/bin/python', '99check_log_files'] *** *** 00_checks/99check_log_files *** Check installation log files for errors, warnings and tracebacks *** *** START TIME: 2016-08-23 18:03:17 *** An error occured while trying to check the installation profile for 'call_master_joinscripts' setting: IOError(2, 'Datei oder Verzeichnis nicht gefunden') Adjusting patterns to ignore respective 'join' messages. The file '/var/log/univention/installation.log' cannot be found, skipping... The file '/var/log/univention/installation.log.gz' cannot be found, skipping... The file '/var/log/univention/installer.log' cannot be found, skipping... The file '/var/log/univention/installer.log.gz' cannot be found, skipping... Errors found in '/var/log/univention/updater.log': E: updater.log:12443, Traceback in univention-upgrade: Traceback (most recent call last): File "/usr/sbin/univention-upgrade", line 453, in do_update update_available = performUpdate(options, checkForUpdates=options.check, silent=False) File "/usr/sbin/univention-upgrade", line 150, in performUpdate update_available_local = func(options, checkForUpdates, silent) File "/usr/sbin/univention-upgrade", line 162, in do_release_update updater = UniventionUpdater() File "/usr/lib/pymodules/python2.7/univention/updater/tools.py", line 577, in __init__ self.architectures = [os.popen('dpkg --print-architecture 2>/dev/null').readline()[:-1]] OSError: [Errno 24] Too many open files E: updater.log:21452, Further information regarding this error: E: updater.log:21453, Error: Could not open file /var/lib/apt/lists/_var_cache_univention-system-setup_packages_._Packages - open (24: Too many open files). E: updater.log:21454, Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/__init__.py", line 189, in call_with_namespace result = self.main(namespace) File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/upgrade_search.py", line 54, in main apps = AppManager.get_all_locally_installed_apps() File "/usr/lib/pymodules/python2.7/univention/appcenter/app.py", line 1449, in get_all_locally_installed_apps if app.is_installed(): File "/usr/lib/pymodules/python2.7/univention/appcenter/app.py", line 911, in is_installed package_manager = AppManager.get_package_manager() File "/usr/lib/pymodules/python2.7/univention/appcenter/app.py", line 1508, in get_package_manager cls._package_manager = PackageManager(lock=False) File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 271, in __init__ self._open_cache() File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 686, in _open_cache self._handle_system_error(*sys.exc_info()) File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 683, in _open_cache _open() File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 672, in _open self.cache = apt.Cache() File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 102, in __init__ self.open(progress) File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 147, in open self._records = apt_pkg.PackageRecords(self._cache) SystemError: Could not initialize package manager. E: updater.log:21478, Further information regarding this error: E: updater.log:21479, Error: Could not open file /var/lib/apt/lists/_var_cache_univention-system-setup_packages_._Packages - open (24: Too many open files). E: updater.log:21480, 2016-08-23 17:43:32,042 ERROR:univention.appcenter.actions.upgrade-search:Could not initialize package manager. E: updater.log:21481, Further information regarding this error: E: updater.log:21482, Error: Could not open file /var/lib/apt/lists/_var_cache_univention-system-setup_packages_._Packages - open (24: Too many open files). E: updater.log:21483, Traceback (most recent call last): Last Traceback "SystemError: Could not initialize package manager." repeats one more time. E: updater.log:21507, Further information regarding this error: E: updater.log:21508, Error: Could not open file /var/lib/apt/lists/_var_cache_univention-system-setup_packages_._Packages - open (24: Too many open files). E: updater.log:21509, Traceback in univention-upgrade: Traceback (most recent call last): File "/usr/sbin/univention-upgrade", line 453, in do_update update_available = performUpdate(options, checkForUpdates=options.check, silent=False) File "/usr/sbin/univention-upgrade", line 150, in performUpdate update_available_local = func(options, checkForUpdates, silent) File "/usr/sbin/univention-upgrade", line 282, in do_app_updates app_upgrade_search.call() File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/__init__.py", line 183, in call return obj.call_with_namespace(namespace) File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/__init__.py", line 189, in call_with_namespace result = self.main(namespace) File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/upgrade_search.py", line 54, in main apps = AppManager.get_all_locally_installed_apps() File "/usr/lib/pymodules/python2.7/univention/appcenter/app.py", line 1449, in get_all_locally_installed_apps if app.is_installed(): File "/usr/lib/pymodules/python2.7/univention/appcenter/app.py", line 911, in is_installed package_manager = AppManager.get_package_manager() File "/usr/lib/pymodules/python2.7/univention/appcenter/app.py", line 1508, in get_package_manager cls._package_manager = PackageManager(lock=False) File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 271, in __init__ self._open_cache() File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 686, in _open_cache self._handle_system_error(*sys.exc_info()) File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 683, in _open_cache _open() File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 672, in _open self.cache = apt.Cache() File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 102, in __init__ self.open(progress) File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 147, in open self._records = apt_pkg.PackageRecords(self._cache) SystemError: Could not initialize package manager. E: updater.log:21542, Further information regarding this error: E: updater.log:21543, Error: Could not open file /var/lib/apt/lists/_var_cache_univention-system-setup_packages_._Packages - open (24: Too many open files). *** END TIME: 2016-08-23 18:03:21 *** *** TEST DURATION (H:MM:SS.ms): 0:00:04.008254 *** *** END *** 1 *** ****************************************************************************** I was able to reproduce it. The tool univention-upgrade runs in an endless loop: ****************************************************************************** root@master411:~# univention-upgrade Starting univention-upgrade. Current UCS version is 4.1-3 errata234 Checking for local repository: none Checking for package updates: found The following packages will be upgraded: univention-directory-manager-tools,python-univention-directory-manager,python-univention-directory-manager-cli,univention-s4-connector,python-univention-connector-s4,univention-errata-level Do you want to continue [Y|n]? y Starting package upgrade done Starting univention-upgrade. Current UCS version is 4.1-3 errata237 Checking for local repository: none Checking for package updates: none Checking for app updates: none Checking for release updates: none Setting update/available root@master411:~# univention-upgrade --updateto=4.1-99 Starting univention-upgrade. Current UCS version is 4.1-3 errata237 Checking for local repository: none Checking for package updates: none Checking for app updates: none Checking for release updates: none root@master411:~# ucr set repository/online/server='https://updates-test.software-univention.de/' Setting repository/online/server File: /etc/apt/sources.list.d/20_ucs-online-component.list File: /etc/apt/mirror.list File: /etc/apt/sources.list.d/15_ucs-online-version.list root@master411:~# univention-upgrade --updateto=4.1-99 Starting univention-upgrade. Current UCS version is 4.1-3 errata237 Checking for local repository: none Checking for package updates: none Checking for app updates: none Checking for release updates: 4.2-0 is available but updater has been instructed to stop at version 4.1-99. root@master411:~# ucr set repository/online/server='https://updates.software-univention.de/' Setting repository/online/server File: /etc/apt/sources.list.d/20_ucs-online-component.list File: /etc/apt/mirror.list File: /etc/apt/sources.list.d/15_ucs-online-version.list root@master411:~# ./activate-errata-test-scope.sh Create repository/online/component/4.1-3-errata-test/description Create repository/online/component/4.1-3-errata-test/version Create repository/online/component/4.1-3-errata-test/server Create repository/online/component/4.1-3-errata-test/username Create repository/online/component/4.1-3-errata-test/password Create repository/online/component/4.1-3-errata-test File: /etc/apt/sources.list.d/20_ucs-online-component.list File: /etc/apt/mirror.list File: /etc/apt/sources.list.d/15_ucs-online-version.list root@master411:~# univention-upgrade --updateto=4.1-99 Starting univention-upgrade. Current UCS version is 4.1-3 errata237 Checking for local repository: none Checking for package updates: found The following packages will be installed: python-mysqldb The following packages will be upgraded: univention-ldap-acl-master,python-univention-appcenter,univention-appcenter,univention-ldap-server,univention-ldap-config,univention-appcenter-docker,univention-directory-listener,univention-ldap-client,univention-management-console-module-updater,univention-updater,univention-management-console-module-appcenter,univention-management-console-module-apps,univention-management-console-module-setup,univention-system-setup Do you want to continue [Y|n]? Starting package upgrade done Starting univention-upgrade. Current UCS version is 4.1-3 errata237 Checking for local repository: none Checking for package updates: none Checking for app updates: none Checking for release updates: none Setting update/available root@master411:~# univention-upgrade --updateto=4.1-99 Starting univention-upgrade. Current UCS version is 4.1-3 errata237 Checking for local repository: none Checking for package updates: none Checking for app updates: none Checking for release updates: none root@master411:~# ucr set repository/online/server='https://updates-test.software-univention.de/' Setting repository/online/server File: /etc/apt/sources.list.d/20_ucs-online-component.list File: /etc/apt/mirror.list File: /etc/apt/sources.list.d/15_ucs-online-version.list root@master411:~# univention-upgrade --updateto=4.1-99 Starting univention-upgrade. Current UCS version is 4.1-3 errata237 Checking for local repository: none Checking for package updates: none Checking for app updates: none Checking for release updates: 4.2-0 is available but updater has been instructed to stop at version 4.1-99. Starting univention-upgrade. Current UCS version is 4.1-3 errata237 Checking for local repository: none Checking for package updates: none Checking for app updates: none Checking for release updates: 4.2-0 is available but updater has been instructed to stop at version 4.1-99. Starting univention-upgrade. Current UCS version is 4.1-3 errata237 Checking for local repository: none Checking for package updates: none Checking for app updates: none Checking for release updates: ^Z [1]+ Angehalten univention-upgrade --updateto=4.1-99 root@master411:~# ******************************************************************************
Fixed in univention-updater 11.0.11-6.1486.201609020926
OK: ucs-test -E dangerous -vvv -s updater OK: ucr set version/patchlevel=2 update/available=no univention-upgrade --check --setucr # Checking for release updates: found: UCS 4.1-3 ucr get update/available # yes OK: ucr set version/patchlevel=2 update/available=no ucr unset repository/online/component/4.1-3-errata sed -i '/^Package: univention-errata-level/,/^$/s/^Version: .*/Version: 1/' /var/lib/dpkg/status univention-upgrade --ignoreterm --ignoressh --noninteractive </dev/null FAIL: ucr set version/patchlevel=2 update/available=no univention-upgrade --ignoreterm --ignoressh --noninteractive --updateto 4.1-2 </dev/null ucr get update/available # yes > if options.updateto and UCS_Version(options.updateto) < UCS_Version(version_next): > dprint(silent, '%s is available but updater has been instructed to stop at version %s.' % (version_next, options.updateto)) >- return False >+ return True that change (and also the later change to "raise UserCancelled()" looks wrong: If the users instructs the Updater to "--updateto 4.1-3" only, they don't want to get told "but there is 4.2-0 available." INHO all following updates should just be ignored as if they did not exist. It's already annoying enough that there is not domain policy to set --updateto everywhere (Bug #37869) FYI: > except SystemExit, e: "e" is unused (In reply to Dirk Wiesenthal from comment #1) > Fixed this by adjusting the return values of the three functions > do_package_updates, do_app_updates, do_release_update RFC: Why are pending app updates no longer considered as updates/available? > Removed the last do_exec(), otherwise I could end up in an infinite loop. OK: do_exec() is already executed after each step. > After playing with it, I think --setucr should be the default, but this has > not been touched. OK: In the past there was a bad interaction with OX, but hopefully that code is no longer used.
As discussed: Reverted in univention-updater 11.0.11-8.1490.201609201429 The functionality may confuse / annoy Administrators that after an explicit --updateto 4.1-3, update/available is set to "yes" because 4.2-0 is available (but unwanted). The problem with Docker will be addressed directly in Docker containers in another bug. One may argue whether or not the behaviour described here is a bug or a feature. I still think that the script should set update/available to "yes" if the next univention-upgrade --check would find the update nonetheless. But as pointed out, the fix would make a lot more sense when supported by some kind of "policy awareness" of univention-upgrade, so that 4.2-0 is not presented as an update if the policy says to stop at 4.1-99. Please verify the revert.
OK: doc/errata/staging/univention-updater.yaml OK: git diff $(git svn find-rev r71622)~1.. -- . OK: git diff $(git svn find-rev r71696)~1.. -- .