Bug 41840 - Set update/available when release updates are available
Set update/available when release updates are available
Status: CLOSED WONTFIX
Product: UCS
Classification: Unclassified
Component: Update - univention-updater
UCS 4.1
Other Linux
: P5 normal (vote)
: UCS 4.1-3-errata
Assigned To: Dirk Wiesenthal
Philipp Hahn
:
Depends on:
Blocks: 41286
  Show dependency treegraph
 
Reported: 2016-07-22 11:27 CEST by Dirk Wiesenthal
Modified: 2016-11-28 14:57 CET (History)
3 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dirk Wiesenthal univentionstaff 2016-07-22 11:27:29 CEST
The appbox image upgrades the container step wise.

If univention-upgrade says that 4.1-1 is available, it upgrades to 4.1-1:

Starting update process, this may take a while.
[...]
Checking for release updates: 4.1-2 is available but updater has been instructed to stop at version 4.1-1.
Checking for package updates:                           found
The following packages will be upgraded:
[...]
Starting package upgrade                                done
Checking for release updates: 4.1-2 is available but updater has been instructed to stop at version 4.1-1.
[...]
Setting update/available

This last line sets update/available=no for some reason. This is a problem because now "univention-app upgrade" stops because this very UCRV is checked before performing an expensive univention-upgrade --check.

This needs to be fixed, otherwise one has to upgrade a container (potentially) multiple times, each time with a 1h delay (or so) until update/available is set back to the correct value (yes).

When fixing this, please have a look at Bug#29642. This is the other way around, but in general update/available needs to be correctly set for Docker.
Comment 1 Dirk Wiesenthal univentionstaff 2016-08-16 00:13:50 CEST
Fixed this by adjusting the return values of the three functions
  do_package_updates, do_app_updates, do_release_update

Removed the last do_exec(), otherwise I could end up in an infinite loop.

Intended to preserve the old functionality of re-executing itself after package/release update.

After playing with it, I think --setucr should be the default, but this has not been touched.

Bug#29642 is very much UMC based and may not be fixable in univention-upgrade itself.

  univention-updater 11.0.11-3.1483.201608160000
Comment 2 Stefan Gohmann univentionstaff 2016-08-17 16:10:38 CEST
A lot of updater tests now fail, see for example 09_updater/23major.test:

http://jenkins.knut.univention.de:8080/job/UCS-4.1/job/UCS-4.1-3/job/AutotestJoin/SambaVersion=s3,Systemrolle=master/19/testReport/09_updater/23major/test/

Please check your changes or adjust the test cases.
Comment 3 Stefan Gohmann univentionstaff 2016-08-17 20:00:18 CEST
From the updater.log:

Traceback (most recent call last):
  File "/usr/sbin/univention-upgrade", line 447, in do_update
    update_available = performUpdate(options, checkForUpdates=options.check, silent=False)
  File "/usr/sbin/univention-upgrade", line 150, in performUpdate
    update_available_local = func(options, checkForUpdates, silent)
  File "/usr/sbin/univention-upgrade", line 281, in do_app_updates
    app_upgrade_search.call_safe()
AttributeError: type object 'UpgradeSearch' has no attribute 'call_safe'

It looks like a dependency between the updater and the app center packages. That is at lest not defined in the YAML files and if possible it should be avoided.
Comment 4 Dirk Wiesenthal univentionstaff 2016-08-18 00:29:12 CEST
Unfortunately, it is not possible to depend on a specific App Center package version in univention-updater. Instead, univention-appcenter depends on univention-updater (classes for handling Components, only legacy code IIRC).

I raise an ImportError() if the univention-appcenter is not present at all. I have adjusted it so that this is raised also when call_safe() is not available. In this case univention-upgrade cannot upgrade Apps.
Comment 5 Dirk Wiesenthal univentionstaff 2016-08-18 00:49:22 CEST
Hmmm... not so sure if this is the fix for the problem. The jenkins logs and my logs differ when I run 09_updater/23major locally. I guess somewhere a UCS 5.0 is mocked up.
Comment 6 Stefan Gohmann univentionstaff 2016-08-18 06:26:10 CEST
(In reply to Dirk Wiesenthal from comment #5)
> Hmmm... not so sure if this is the fix for the problem. The jenkins logs and
> my logs differ when I run 09_updater/23major locally. I guess somewhere a
> UCS 5.0 is mocked up.

It is not only the 23major test case. Seven test cases on all roles fail now:

 09_updater.20patch-single.test
 09_updater.21patch-multiple.test
 09_updater.22minor.test
 09_updater.23major.test
 09_updater.27updateto.test
 09_updater.34updater-check.test
 09_updater.49component-version-current-dep.test

For me it is reproducible:
------------------------------------------------------------------------------
root@master411:~# /usr/share/ucs-test/09_updater/23major -f
6769 blocks
gpg: keyring `/root/.gnupg/secring.gpg' created
gpg: keyring `/root/.gnupg/pubring.gpg' created
OK
Stopping periodic command scheduler: cron.
dpkg-preconfigure: kann Stdin nicht wieder öffnen: Datei oder Verzeichnis nicht gefunden
Vormals nicht ausgewähltes Paket test-27201-26254 wird gewählt.
(Lese Datenbank ... 101755 Dateien und Verzeichnisse sind derzeit installiert.)
Entpacken von test-27201-26254 (aus .../test-27201-26254_1_amd64.deb) ...
test-27201-26254 (1) wird eingerichtet ...
Checking network repository
Update to = 4.1-1
Checking network repository
Update to = 4.2-0
Checking network repository
Update to = 5.0-0
Checking network repository
System is up to date (UCS 5.0-0)
=== RESULT: 100 ===
Starting 1 ucs-test at 2015-11-18 13:23:18 to /dev/null
Update localhost repository one major version............................................................................................................... Test passed
root@master411:~# ./activate-errata-test-scope.sh 
Create repository/online/component/4.1-3-errata-test/description
Create repository/online/component/4.1-3-errata-test/version
Create repository/online/component/4.1-3-errata-test/server
Create repository/online/component/4.1-3-errata-test/username
Create repository/online/component/4.1-3-errata-test/password
Create repository/online/component/4.1-3-errata-test
File: /etc/apt/sources.list.d/20_ucs-online-component.list
File: /etc/apt/mirror.list
File: /etc/apt/sources.list.d/15_ucs-online-version.list
root@master411:~# univention-upgrade 

Starting univention-upgrade. Current UCS version is 4.1-3 errata234

Checking for local repository:                          none
Checking for package updates:                           found

The following packages will be installed:
 python-mysqldb
The following packages will be upgraded:
 univention-directory-manager-tools,python-univention-directory-manager,python-univention-directory-manager-cli,python-univention-appcenter,univention-s4-connector,python-univention-connector-s4,univention-appcenter,univention-appcenter-docker,univention-management-console-module-updater,univention-updater,univention-management-console-module-appcenter,univention-management-console-module-apps

Do you want to continue [Y|n]? y

Starting package upgrade                                done

Starting univention-upgrade. Current UCS version is 4.1-3 errata234

Checking for local repository:                          none
Checking for package updates:                           none
Checking for app updates:                               none
Checking for release updates:                           none
root@master411:~# /usr/share/ucs-test/09_updater/23major -f
6769 blocks
gpg: keyring `/root/.gnupg/secring.gpg' created
gpg: keyring `/root/.gnupg/pubring.gpg' created
OK
Stopping periodic command scheduler: cron.
dpkg-preconfigure: kann Stdin nicht wieder öffnen: Datei oder Verzeichnis nicht gefunden
Vormals nicht ausgewähltes Paket test-29434-24821 wird gewählt.
(Lese Datenbank ... 101836 Dateien und Verzeichnisse sind derzeit installiert.)
Entpacken von test-29434-24821 (aus .../test-29434-24821_1_amd64.deb) ...
test-29434-24821 (1) wird eingerichtet ...
Checking network repository
Update to = 4.1-1
Checking network repository
Update to = 4.2-0
Checking network repository
Update to = 5.0-0
Checking network repository
System is up to date (UCS 5.0-0)
**************** Test failed above this line ****************
ERROR 23major:50 0
ERROR test -n "$(ucr get update/available)"
=== RESULT: 110 ===
Starting 1 ucs-test at 2015-11-18 13:25:21 to /dev/null
Update localhost repository one major version............................................................................................................... Test failed
root@master411:~# dpkg -l | grep univention-updater
ii  univention-updater                                  11.0.11-4.1484.201608172305                  all          UCS - system upgrading tool
root@master411:~# 
------------------------------------------------------------------------------

(In reply to Dirk Wiesenthal from comment #4)
> Unfortunately, it is not possible to depend on a specific App Center package
> version in univention-updater. Instead, univention-appcenter depends on
> univention-updater (classes for handling Components, only legacy code IIRC).
> 
> I raise an ImportError() if the univention-appcenter is not present at all.
> I have adjusted it so that this is raised also when call_safe() is not
> available. In this case univention-upgrade cannot upgrade Apps.

That means if we release the univention-updater package no one can install app updates via univention-upgrade until we release the appcenter package. That is for me a blocker. Why can't we check on code level if the function is available and use the old one if it is not available?
Comment 7 Dirk Wiesenthal univentionstaff 2016-08-19 00:31:27 CEST
Okay, using old code, not depending on univention-appcenter update anymore.

Got all the tests running (yesterday, I had a repo registered which caused the tests to fail before the interesting part). Fixed in
  univention-updater 11.0.11-5.1485.201608190011
Comment 8 Stefan Gohmann univentionstaff 2016-08-24 07:37:25 CEST
It looks like a regression was introduced by these changes. The AD member mode Jenkins jobs uses updates-test.software-univention.de. Since a few days UCS 4.2 is available on updates-test. Anyway, the test case 00_checks.99check_log_files found the following entries:

******************************************************************************
*** BEGIN *** ['/usr/bin/python', '99check_log_files'] ***
*** 00_checks/99check_log_files *** Check installation log files for errors, warnings and tracebacks ***
*** START TIME: 2016-08-23 18:03:17 ***

An error occured while trying to check the installation profile for 'call_master_joinscripts' setting: IOError(2, 'Datei oder Verzeichnis nicht gefunden') Adjusting patterns to ignore respective 'join' messages.

The file '/var/log/univention/installation.log' cannot be found, skipping...

The file '/var/log/univention/installation.log.gz' cannot be found, skipping...

The file '/var/log/univention/installer.log' cannot be found, skipping...

The file '/var/log/univention/installer.log.gz' cannot be found, skipping...

Errors found in '/var/log/univention/updater.log':

 E: updater.log:12443, Traceback in univention-upgrade:
Traceback (most recent call last):
  File "/usr/sbin/univention-upgrade", line 453, in do_update
    update_available = performUpdate(options, checkForUpdates=options.check, silent=False)
  File "/usr/sbin/univention-upgrade", line 150, in performUpdate
    update_available_local = func(options, checkForUpdates, silent)
  File "/usr/sbin/univention-upgrade", line 162, in do_release_update
    updater = UniventionUpdater()
  File "/usr/lib/pymodules/python2.7/univention/updater/tools.py", line 577, in __init__
    self.architectures = [os.popen('dpkg --print-architecture 2>/dev/null').readline()[:-1]]
OSError: [Errno 24] Too many open files

 E: updater.log:21452, Further information regarding this error:
 E: updater.log:21453, Error: Could not open file /var/lib/apt/lists/_var_cache_univention-system-setup_packages_._Packages - open (24: Too many open files).
 E: updater.log:21454, Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/__init__.py", line 189, in call_with_namespace
    result = self.main(namespace)
  File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/upgrade_search.py", line 54, in main
    apps = AppManager.get_all_locally_installed_apps()
  File "/usr/lib/pymodules/python2.7/univention/appcenter/app.py", line 1449, in get_all_locally_installed_apps
    if app.is_installed():
  File "/usr/lib/pymodules/python2.7/univention/appcenter/app.py", line 911, in is_installed
    package_manager = AppManager.get_package_manager()
  File "/usr/lib/pymodules/python2.7/univention/appcenter/app.py", line 1508, in get_package_manager
    cls._package_manager = PackageManager(lock=False)
  File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 271, in __init__
    self._open_cache()
  File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 686, in _open_cache
    self._handle_system_error(*sys.exc_info())
  File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 683, in _open_cache
    _open()
  File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 672, in _open
    self.cache = apt.Cache()
  File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 102, in __init__
    self.open(progress)
  File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 147, in open
    self._records = apt_pkg.PackageRecords(self._cache)
SystemError: Could not initialize package manager.

 E: updater.log:21478, Further information regarding this error:
 E: updater.log:21479, Error: Could not open file /var/lib/apt/lists/_var_cache_univention-system-setup_packages_._Packages - open (24: Too many open files).
 E: updater.log:21480, 2016-08-23 17:43:32,042 ERROR:univention.appcenter.actions.upgrade-search:Could not initialize package manager.
 E: updater.log:21481, Further information regarding this error:
 E: updater.log:21482, Error: Could not open file /var/lib/apt/lists/_var_cache_univention-system-setup_packages_._Packages - open (24: Too many open files).
 E: updater.log:21483, Traceback (most recent call last):
Last Traceback "SystemError: Could not initialize package manager." repeats one more time.

 E: updater.log:21507, Further information regarding this error:
 E: updater.log:21508, Error: Could not open file /var/lib/apt/lists/_var_cache_univention-system-setup_packages_._Packages - open (24: Too many open files).
 E: updater.log:21509, Traceback in univention-upgrade:
Traceback (most recent call last):
  File "/usr/sbin/univention-upgrade", line 453, in do_update
    update_available = performUpdate(options, checkForUpdates=options.check, silent=False)
  File "/usr/sbin/univention-upgrade", line 150, in performUpdate
    update_available_local = func(options, checkForUpdates, silent)
  File "/usr/sbin/univention-upgrade", line 282, in do_app_updates
    app_upgrade_search.call()
  File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/__init__.py", line 183, in call
    return obj.call_with_namespace(namespace)
  File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/__init__.py", line 189, in call_with_namespace
    result = self.main(namespace)
  File "/usr/lib/pymodules/python2.7/univention/appcenter/actions/upgrade_search.py", line 54, in main
    apps = AppManager.get_all_locally_installed_apps()
  File "/usr/lib/pymodules/python2.7/univention/appcenter/app.py", line 1449, in get_all_locally_installed_apps
    if app.is_installed():
  File "/usr/lib/pymodules/python2.7/univention/appcenter/app.py", line 911, in is_installed
    package_manager = AppManager.get_package_manager()
  File "/usr/lib/pymodules/python2.7/univention/appcenter/app.py", line 1508, in get_package_manager
    cls._package_manager = PackageManager(lock=False)
  File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 271, in __init__
    self._open_cache()
  File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 686, in _open_cache
    self._handle_system_error(*sys.exc_info())
  File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 683, in _open_cache
    _open()
  File "/usr/lib/pymodules/python2.7/univention/lib/package_manager.py", line 672, in _open
    self.cache = apt.Cache()
  File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 102, in __init__
    self.open(progress)
  File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 147, in open
    self._records = apt_pkg.PackageRecords(self._cache)
SystemError: Could not initialize package manager.

 E: updater.log:21542, Further information regarding this error:
 E: updater.log:21543, Error: Could not open file /var/lib/apt/lists/_var_cache_univention-system-setup_packages_._Packages - open (24: Too many open files).
*** END TIME: 2016-08-23 18:03:21 ***
*** TEST DURATION (H:MM:SS.ms): 0:00:04.008254 ***
*** END *** 1 ***
******************************************************************************

I was able to reproduce it. The tool univention-upgrade runs in an endless loop:
******************************************************************************
root@master411:~# univention-upgrade

Starting univention-upgrade. Current UCS version is 4.1-3 errata234

Checking for local repository:                          none
Checking for package updates:                           found

The following packages will be upgraded:
 univention-directory-manager-tools,python-univention-directory-manager,python-univention-directory-manager-cli,univention-s4-connector,python-univention-connector-s4,univention-errata-level

Do you want to continue [Y|n]? y

Starting package upgrade                                done

Starting univention-upgrade. Current UCS version is 4.1-3 errata237

Checking for local repository:                          none
Checking for package updates:                           none
Checking for app updates:                               none
Checking for release updates:                           none
Setting update/available
root@master411:~# univention-upgrade --updateto=4.1-99

Starting univention-upgrade. Current UCS version is 4.1-3 errata237

Checking for local repository:                          none
Checking for package updates:                           none
Checking for app updates:                               none
Checking for release updates:                           none
root@master411:~# ucr set repository/online/server='https://updates-test.software-univention.de/'
Setting repository/online/server
File: /etc/apt/sources.list.d/20_ucs-online-component.list
File: /etc/apt/mirror.list
File: /etc/apt/sources.list.d/15_ucs-online-version.list
root@master411:~# univention-upgrade --updateto=4.1-99

Starting univention-upgrade. Current UCS version is 4.1-3 errata237

Checking for local repository:                          none
Checking for package updates:                           none
Checking for app updates:                               none
Checking for release updates:                           4.2-0 is available but updater has been instructed to stop at version 4.1-99.
root@master411:~# ucr set repository/online/server='https://updates.software-univention.de/'
Setting repository/online/server
File: /etc/apt/sources.list.d/20_ucs-online-component.list
File: /etc/apt/mirror.list
File: /etc/apt/sources.list.d/15_ucs-online-version.list
root@master411:~# ./activate-errata-test-scope.sh
Create repository/online/component/4.1-3-errata-test/description
Create repository/online/component/4.1-3-errata-test/version
Create repository/online/component/4.1-3-errata-test/server
Create repository/online/component/4.1-3-errata-test/username
Create repository/online/component/4.1-3-errata-test/password
Create repository/online/component/4.1-3-errata-test
File: /etc/apt/sources.list.d/20_ucs-online-component.list
File: /etc/apt/mirror.list
File: /etc/apt/sources.list.d/15_ucs-online-version.list
root@master411:~# univention-upgrade --updateto=4.1-99

Starting univention-upgrade. Current UCS version is 4.1-3 errata237

Checking for local repository:                          none
Checking for package updates:                           found

The following packages will be installed:
 python-mysqldb
The following packages will be upgraded:
 univention-ldap-acl-master,python-univention-appcenter,univention-appcenter,univention-ldap-server,univention-ldap-config,univention-appcenter-docker,univention-directory-listener,univention-ldap-client,univention-management-console-module-updater,univention-updater,univention-management-console-module-appcenter,univention-management-console-module-apps,univention-management-console-module-setup,univention-system-setup

Do you want to continue [Y|n]?

Starting package upgrade                                done

Starting univention-upgrade. Current UCS version is 4.1-3 errata237

Checking for local repository:                          none
Checking for package updates:                           none
Checking for app updates:                               none
Checking for release updates:                           none
Setting update/available
root@master411:~# univention-upgrade --updateto=4.1-99

Starting univention-upgrade. Current UCS version is 4.1-3 errata237

Checking for local repository:                          none
Checking for package updates:                           none
Checking for app updates:                               none
Checking for release updates:                           none
root@master411:~# ucr set repository/online/server='https://updates-test.software-univention.de/'
Setting repository/online/server
File: /etc/apt/sources.list.d/20_ucs-online-component.list
File: /etc/apt/mirror.list
File: /etc/apt/sources.list.d/15_ucs-online-version.list
root@master411:~# univention-upgrade --updateto=4.1-99

Starting univention-upgrade. Current UCS version is 4.1-3 errata237

Checking for local repository:                          none
Checking for package updates:                           none
Checking for app updates:                               none
Checking for release updates:                           4.2-0 is available but updater has been instructed to stop at version 4.1-99.

Starting univention-upgrade. Current UCS version is 4.1-3 errata237

Checking for local repository:                          none
Checking for package updates:                           none
Checking for app updates:                               none
Checking for release updates:                           4.2-0 is available but updater has been instructed to stop at version 4.1-99.

Starting univention-upgrade. Current UCS version is 4.1-3 errata237

Checking for local repository:                          none
Checking for package updates:                           none
Checking for app updates:                               none
Checking for release updates:                          ^Z
[1]+  Angehalten              univention-upgrade --updateto=4.1-99
root@master411:~#
******************************************************************************
Comment 9 Dirk Wiesenthal univentionstaff 2016-09-02 09:38:38 CEST
Fixed in
  univention-updater 11.0.11-6.1486.201609020926
Comment 10 Philipp Hahn univentionstaff 2016-09-16 16:49:07 CEST
OK: ucs-test -E dangerous -vvv -s updater

OK:
 ucr set version/patchlevel=2 update/available=no
 univention-upgrade --check --setucr
 # Checking for release updates:                           found: UCS 4.1-3
 ucr get update/available
 # yes

OK:
 ucr set version/patchlevel=2 update/available=no
 ucr unset repository/online/component/4.1-3-errata
 sed -i '/^Package: univention-errata-level/,/^$/s/^Version: .*/Version: 1/'  /var/lib/dpkg/status
 univention-upgrade --ignoreterm --ignoressh --noninteractive </dev/null

FAIL:
 ucr set version/patchlevel=2 update/available=no
 univention-upgrade --ignoreterm --ignoressh --noninteractive --updateto 4.1-2 </dev/null
 ucr get update/available
 # yes
>     if options.updateto and UCS_Version(options.updateto) < UCS_Version(version_next):
>         dprint(silent, '%s is available but updater has been instructed to stop at version %s.' % (version_next, options.updateto))
>-        return False
>+        return True
that change (and also the later change to "raise UserCancelled()" looks wrong:
If the users instructs the Updater to "--updateto 4.1-3" only, they don't want to get told "but there is 4.2-0 available."
INHO all following updates should just be ignored as if they did not exist.
It's already annoying enough that there is not domain policy to set --updateto everywhere (Bug #37869)

FYI:
>         except SystemExit, e:
"e" is unused

(In reply to Dirk Wiesenthal from comment #1)
> Fixed this by adjusting the return values of the three functions
>   do_package_updates, do_app_updates, do_release_update

RFC: Why are pending app updates no longer considered as updates/available?

> Removed the last do_exec(), otherwise I could end up in an infinite loop.

OK: do_exec() is already executed after each step.

> After playing with it, I think --setucr should be the default, but this has
> not been touched.

OK: In the past there was a bad interaction with OX, but hopefully that code is no longer used.
Comment 11 Dirk Wiesenthal univentionstaff 2016-09-20 14:43:29 CEST
As discussed: Reverted in
  univention-updater 11.0.11-8.1490.201609201429

The functionality may confuse / annoy Administrators that after an explicit --updateto 4.1-3, update/available is set to "yes" because 4.2-0 is available (but unwanted).

The problem with Docker will be addressed directly in Docker containers in another bug.

One may argue whether or not the behaviour described here is a bug or a feature. I still think that the script should set update/available to "yes" if the next univention-upgrade --check would find the update nonetheless. But as pointed out, the fix would make a lot more sense when supported by some kind of "policy awareness" of univention-upgrade, so that 4.2-0 is not presented as an update if the policy says to stop at 4.1-99.

Please verify the revert.
Comment 12 Philipp Hahn univentionstaff 2016-09-21 13:31:11 CEST
OK: doc/errata/staging/univention-updater.yaml
OK: git diff $(git svn find-rev r71622)~1.. -- .
OK: git diff $(git svn find-rev r71696)~1.. -- .