Bug 53228 - Fix univention-app update-check
Fix univention-app update-check
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: App Center
UCS 4.4
Other Linux
: P5 normal (vote)
: UCS 4.4-8-errata
Assigned To: Philipp Hahn
Felix Botner
:
Depends on: 53297 52771
Blocks: 53173 53429
  Show dependency treegraph
 
Reported: 2021-05-05 17:49 CEST by Philipp Hahn
Modified: 2021-06-22 13:49 CEST (History)
5 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 4: Will affect most installed domains
How will those affected feel about the bug?: 5: Blocking further progress on the daily work
User Pain: 0.571
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review: Yes
Ticket number:
Bug group (optional):
Max CVSS v3 score:
hahn: Patch_Available+


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Philipp Hahn univentionstaff 2021-05-05 17:49:15 CEST
+++ This bug was initially created as a clone of Bug #52771 +++

Updating 4.4-8 to 5.0-0 via UMC fails *if* a component App is installed which is not available for UCS-5, e.g. Pykota or Nagios-Server. The UMC exception is:

> Ein Fehler ist aufgetreten:
> Die Anfrage konnte nicht ausgeführt werden.
>
> Fehlernachricht des Servers:
>
> Fehler beim Verbindungsaufbau zum Update-Server. Bitte überprüfen Sie die Proxy- oder Firewall-Einstellungen, falls vorhanden. Es kann sich sich auch um ein Problem des konfigurierten DNS-Servers handeln. Dies ist die Fehlermeldung: 'NoneType' object has no attribute 'update'

Adding debug code to umc/python/updater/__init__.py provides more detail:

> Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/univention/management/console/modules/updater/__init__.py", line 456, in status
>     blocking_components.update(set(blocking_apps))
> AttributeError: 'NoneType' object has no attribute 'update'


python2 -c 'from univention.updater.tools import UniventionUpdater as U;u=U();print(u.get_all_available_release_updates())'
> (['5.0-0'], None)

univention-app update-check --ucs-version 5.0-0
> …
> The update to 5.0-0 is currently not possible, because the following Apps are not available for UCS 5.0-0:
>  * Print server Quota (pykota)
Comment 1 Erik Damrose univentionstaff 2021-05-05 18:06:28 CEST
The impact is significant for UMC users:

When opening the UMC updater module the following non helpful error is shown:

"
Error contacting the update server. Please check your proxy or firewall settings, if any. Or it may be a problem with your configured DNS server. This is the error message: 'NoneType' object has no attribute 'update'
"

After confirming the message, the updater module does not finish loading, updates can not be installed via UMC. There is no way to show any additional info at this point.

This was introduced in bug 52771, released as UCS 4.4e909 on 2021-03-10
Comment 2 Philipp Hahn univentionstaff 2021-05-05 18:21:33 CEST
TEST: umc-command -U Administrator -P univention updater/updates/get

Needs to be released for the following scopes:
- errata4.4-7
- errata4.4-8
- ucs4.4-8 (probably, needs more thinking)
Comment 3 Philipp Hahn univentionstaff 2021-05-05 18:35:29 CEST
- [ ] For errata4.4-8: <https://git.knut.univention.de/univention/ucs/-/merge_requests/91>

- [ ] For UCS-5
Comment 4 Philipp Hahn univentionstaff 2021-05-05 18:44:08 CEST
[4.4-8] e364ba804b Bug #53228 up: Handle non-blocking components
[4.4-8] 6e6fa8287e fix[updater]: Handle non-blocking components
 base/univention-updater/debian/changelog               | 6 ++++++
 base/univention-updater/umc/python/updater/__init__.py | 4 ++--
 2 files changed, 8 insertions(+), 2 deletions(-)

[4.4-8] d5f724823a feat[updater]: Improve exception handling
 base/univention-updater/umc/python/updater/__init__.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

[4.4-8] 7d41a54391 style[updater]: Remove duplicate exception
 base/univention-updater/umc/python/updater/__init__.py | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

[4.4-8] 2df52cc9f2 style[updater]: Remove dead code
 base/univention-updater/umc/python/updater/__init__.py | 2 --
 1 file changed, 2 deletions(-)

Package: univention-updater
Version: 14.0.3-11A~4.4.0.202105051841
Branch: ucs_4.4-0
Scope: errata4.4-8

[4.4-8] c396613c8b Bug #53228: univention-updater 14.0.3-11A~4.4.0.202105051841
 doc/errata/staging/univention-updater.yaml | 11 +++++++++++
 1 file changed, 11 insertions(+)
Comment 5 Philipp Hahn univentionstaff 2021-05-06 09:24:52 CEST
Thoughts from tonight: We should add (upgrade) tests via UMC, best through Selenium (or Helenium) Browser Tests, but at least through `umc-command` to trigger the UMC path. Currently we only have CLI tests in ucs-test/tests/09_updater/
This requires more work as AppC has to be mocked/faked, too. For that take a look at ucs-test/tests/20_appcenter/appcentertest.py
Comment 6 Philipp Hahn univentionstaff 2021-05-06 11:55:32 CEST
Usability feedback: The UMC Updater module shows some larger text:

> Weitere Releaseupdates sind verfügbar, können jedoch nicht installiert werden. Die momentan installierte Version der Applikation Druckserver Quota (pykota) ist nicht verfügbar für alle neueren UCS Releases. Sie können auf die Veröffentlichung der App für die neue UCS Version warten. Sie können auch im Modul "App Center" nach Alternativen suchen oder die Applikation deinstallieren.

It is easy to overlook as it uses the same black text color and font size and has no warning sign. The real issue "pykota" is also hard to find in this long text block.

Even worse the button "Release-Aktualisierung installieren" is still enabled.
Click on it will start the upgrade, which is then aborted by "preup.sh" executing "check.sh".

My expectations are:
- show a big fat warning sign
- list the blocking components in a HTML list
- disable the upgrade button
- maybe add a checkbox with "[ ] yes I understand the problem but I still want to try the upgrade knowing that Univention support will not help me while I ignore the underlying problem."
Comment 7 Philipp Hahn univentionstaff 2021-05-06 13:00:31 CEST
As discussed: We're renaming the /releases.json to /ucs-releases.json to prevent current 4.4-[78] systems from ever finding the 5.0-0 release.

[4.4-8] 6c2946c90b fix[up]: Rename to ucs-releases.json
 base/univention-updater/modules/univention/updater/tools.py | 2 +-
 base/univention-updater/tests/mockups.py                    | 2 +-
 base/univention-updater/tests/test_updater.py               | 6 +++---
 test/ucs-test/tests/09_updater/create_releases_json.py      | 4 ++--
 4 files changed, 7 insertions(+), 7 deletions(-)

[4.4-8] 14a806582c fix[up]: Rename to ucs-releases.json
 base/univention-updater/debian/changelog | 6 ++++++
 test/ucs-test/debian/changelog           | 6 ++++++
 2 files changed, 12 insertions(+)

Package: ucs-test
Version: 9.0.7-32A~4.4.0.202105061236

Package: univention-updater
Version: 14.0.3-12A~4.4.0.202105061239
Branch: ucs_4.4-0
Scope: errata4.4-8

[4.4-8] 64a3714d40 Bug #53228: univention-updater 14.0.3-12A~4.4.0.202105061239
 doc/errata/staging/univention-updater.yaml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

FYI: errata will be releases for 4.4-8 *only* to prevent upgrade from 4.4-7.
Comment 8 Philipp Hahn univentionstaff 2021-05-06 13:29:24 CEST
Same change slightly adapted for and merged into 5.0-0: f6ea45990f..44aca4d878

Package: univention-updater
Version: 15.0.3-54A~5.0.0.202105061315
Branch: ucs_5.0-0

Package: ucs-test
Version: 10.0.5-15A~5.0.0.202105061318
Comment 9 Felix Botner univentionstaff 2021-05-06 17:13:12 CEST
TODO jenkins test

OK - releases.json -> ucs-releases.json
OK - ucs-test
OK - finds update with ucs-releases.json 
OK - and no update with releases.json
OK - update with non existing components is "blocked"
OK - yaml
OK - merge to 5.0
Comment 10 Felix Botner univentionstaff 2021-05-06 17:56:52 CEST
in UCS5 the file is required 

/usr/lib/python3/dist-packages/univention/updater/scripts/upgrade.py
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/univention/updater/scripts/upgrade.py", line 480, in <module>
    main()
  File "/usr/lib/python3/dist-packages/univention/updater/scripts/upgrade.py", line 419, in main
    do_update(options)
  File "/usr/lib/python3/dist-packages/univention/updater/scripts/upgrade.py", line 441, in do_update
    update_available = performUpdate(options, checkForUpdates=options.check, silent=False)
  File "/usr/lib/python3/dist-packages/univention/updater/scripts/upgrade.py", line 139, in performUpdate
    if func(options, checkForUpdates, silent):
  File "/usr/lib/python3/dist-packages/univention/updater/scripts/upgrade.py", line 192, in do_package_updates
    updater = UniventionUpdater()
  File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1241, in __init__
    self.ucr_reinit()
  File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1300, in ucr_reinit
    self._get_releases()
  File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line 1325, in _get_releases
    raise ConfigurationError(uri, 'non-existing prefix "%s": %s' % (self.repourl.path, uri))
univention.updater.errors.ConfigurationError: Configuration error: non-existing prefix "": http://updates-test.software-univention.de/ucs-releases.json

is this OK?
Comment 11 Philipp Hahn univentionstaff 2021-05-07 14:44:16 CEST
(In reply to Felix Botner from comment #10)
> in UCS5 the file is required 
...
>     self.ucr_reinit()
>   File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line
> 1300, in ucr_reinit
>     self._get_releases()
>   File "/usr/lib/python3/dist-packages/univention/updater/tools.py", line
> 1325, in _get_releases
>     raise ConfigurationError(uri, 'non-existing prefix "%s": %s' %
> (self.repourl.path, uri))
> univention.updater.errors.ConfigurationError: Configuration error:
> non-existing prefix "":
> http://updates-test.software-univention.de/ucs-releases.json
> 
> is this OK?

Yes, this is intended: A UCS-5 mirror must include that file and it is a fatal "ConfigurationError" if it cannot be found or accessed. Either
- repository/online/server
- repository/online/port
- repository/online/prefix
- proxy/http*
- ...
is wrong, which requires manual intervention.
Another case would be a (transient) network error, which we also cannot fix automatically and required manual investigation by the administrator.
Comment 12 Erik Damrose univentionstaff 2021-05-10 12:45:07 CEST
If the file is required before we want UCS to install any package updates, the current behavior is inconsistent. The CLI updater fails with  

Starting univention-upgrade. Current UCS version is 5.0-0 errata0

Checking for local repository:                          none
The connection to the repository server failed: Configuration error: non-existing prefix "": http://updates.software-univention.de/ucs-releases.json. Please check the repository configuration and the network connection.

No package updates can be installed. But when using the UMC updater module, i get no error, no popup warning of any misconfiguration. The backend checks for package updates and if any are found, installs them without issues.
Comment 13 Philipp Hahn univentionstaff 2021-05-10 15:11:59 CEST
(In reply to Erik Damrose from comment #12)
> If the file is required before we want UCS to install any package updates,
> the current behavior is inconsistent. The CLI updater fails with  
> 
> Starting univention-upgrade. Current UCS version is 5.0-0 errata0
> 
> Checking for local repository:                          none
> The connection to the repository server failed: Configuration error:
> non-existing prefix "":
> http://updates.software-univention.de/ucs-releases.json. Please check the
> repository configuration and the network connection.
> 
> No package updates can be installed.

By setting UCRV "repository/online/server=https://updates.software-univention.de/" the file /etc/apt/sources.list.d/15_ucs-online-version.list becomes invalid: it no longer contains the required "deb"-lines but Tracebacks.

`univention-upgrade` checks for that is fails as expected.


> But when using the UMC updater module,
> I get no error, no popup warning of any misconfiguration. The backend checks
> for package updates and if any are found, installs them without issues.

This is already broken in UCS-4, so no UCS-5 regression:
- You can enter an invalid URL in UCM module "repository setting" and it is *not* detected
- The updater module still shows the "Check for package updates" even when `grep ^[^#] /etc/apt/sources.list.d/*` is empty

If you care create a new bug to improve and fix UMC.
Comment 15 Felix Botner univentionstaff 2021-05-12 11:01:58 CEST
OK -  jenkins test