Bug 51598 - univention-appcenter: migrate lib to python3
univention-appcenter: migrate lib to python3
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: App Center
UCS 5.0
Other Linux
: P5 normal (vote)
: UCS 5.0
Assigned To: Dirk Wiesenthal
Florian Best
: interim-1
: 51326 53133 (view as bug list)
Depends on:
Blocks: 51326 52018
  Show dependency treegraph
 
Reported: 2020-07-02 11:14 CEST by Dirk Wiesenthal
Modified: 2023-06-27 15:34 CEST (History)
3 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dirk Wiesenthal univentionstaff 2020-07-02 11:14:11 CEST
... and change shebang in /usr/bin/univention-app, I guess
Comment 1 Felix Botner univentionstaff 2020-07-16 09:49:30 CEST
please fix 8985bf8565b2d353b41f404c829180d5e506863c, this breaks the test 20_appcenter.11_check_apps_ini_field_values.slave098

Traceback (most recent call last):
(2020-07-16 01:56:11.590746)   File "11_check_apps_ini_field_values", line 88, in <module>
(2020-07-16 01:56:11.590829)     check_ini_files()
(2020-07-16 01:56:11.590926) TypeError: check_ini_files() takes exactly 1 argument (0 given)
Comment 2 Felix Botner univentionstaff 2020-07-16 15:09:32 CEST
This also fails in the app test, please fix asap, and why is this committed to 4.4-5 with target milestone 5.0-0
Comment 3 Florian Best univentionstaff 2020-09-17 14:26:16 CEST
Traceback (most recent call last):
  File "<stdin>", line 62, in <module>
  File "/usr/lib/python3/dist-packages/univention/appcenter/app_cache.py", line 79, in get_all_locally_installed_apps
    for app in self.get_every_single_app():
  File "/usr/lib/python3/dist-packages/univention/appcenter/app_cache.py", line 543, in get_every_single_app
    for app in app_cache.get_every_single_app():
  File "/usr/lib/python3/dist-packages/univention/appcenter/app_cache.py", line 502, in get_every_single_app
    ret.extend(app_cache.get_every_single_app())
  File "/usr/lib/python3/dist-packages/univention/appcenter/app_cache.py", line 349, in get_every_single_app
    if _cmp_mtimes(archive_modified, self._cache_modified_mtime) == 1:
  File "/usr/lib/python3/dist-packages/univention/appcenter/app_cache.py", line 62, in _cmp_mtimes
    return cmp(mtime1, mtime2)
NameError: name 'cmp' is not defined
Comment 4 Florian Best univentionstaff 2020-09-29 20:31:55 CEST
Traceback (most recent call last):
  File "<stdin>", line 62, in <module>
  File "/usr/lib/python3/dist-packages/univention/appcenter/app_cache.py", line 80, in get_all_locally_installed_apps
    for app in self.get_every_single_app():
  File "/usr/lib/python3/dist-packages/univention/appcenter/app_cache.py", line 544, in get_every_single_app
    for app in app_cache.get_every_single_app():
  File "/usr/lib/python3/dist-packages/univention/appcenter/app_cache.py", line 503, in get_every_single_app
    ret.extend(app_cache.get_every_single_app())
  File "/usr/lib/python3/dist-packages/univention/appcenter/app_cache.py", line 360, in get_every_single_app
    app = self._build_app_from_ini(ini)
  File "/usr/lib/python3/dist-packages/univention/appcenter/app_cache.py", line 312, in _build_app_from_ini
    attr.post_creation(app)
  File "/usr/lib/python3/dist-packages/univention/appcenter/app.py", line 397, in post_creation
    value = ''.join(fd.readlines()).strip()
TypeError: sequence item 0: expected str instance, bytes found
Comment 5 Florian Best univentionstaff 2020-09-29 20:33:12 CEST
(In reply to Florian Best from comment #3)
> NameError: name 'cmp' is not defined

Alternative something like this?:
def _cmp_mtimes(mtime1, mtime2):
    mtime1 = float('{:.3f}'.format(mtime1)) if mtime1 is not None else 0.0
    mtime2 = float('{:.3f}'.format(mtime2)) if mtime2 is not None else 0.0
    return 0 if mtime1 == mtime2 else (-1 if mtime1 > mtime2 else 1)
Comment 6 Florian Best univentionstaff 2021-02-02 10:57:31 CET
[2021-02-02 10:20:00.024858] Traceback (most recent call last):
[2021-02-02 10:20:00.024895]   File "/usr/lib/python3/dist-packages/univention/management/console/base.py", line 359, in __error_handling
[2021-02-02 10:20:00.024919]     six.reraise(etype, exc, etraceback)
[2021-02-02 10:20:00.024943]   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
[2021-02-02 10:20:00.024964]     raise value
[2021-02-02 10:20:00.024989]   File "/usr/lib/python3/dist-packages/univention/management/console/base.py", line 262, in execute
[2021-02-02 10:20:00.025012]     function.__func__(self, request, *args, **kwargs)
[2021-02-02 10:20:00.025038]   File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 321, in _response
[2021-02-02 10:20:00.025062]     result = _multi_response(self, request)
[2021-02-02 10:20:00.025087]   File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 181, in _response
[2021-02-02 10:20:00.025110]     return function(self, request)
[2021-02-02 10:20:00.025189]   File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 443, in _response
[2021-02-02 10:20:00.025215]     return list(function(self, iterator, *nones))
[2021-02-02 10:20:00.025241]   File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 289, in _fake_func
[2021-02-02 10:20:00.025263]     yield function(self, *args)
[2021-02-02 10:20:00.025290]   File "/usr/lib/python3/dist-packages/univention/management/console/modules/appcenter/__init__.py", line 200, in version
[2021-02-02 10:20:00.025313]     if not info.is_compatible(version):
[2021-02-02 10:20:00.025339]   File "/usr/lib/python3/dist-packages/univention/appcenter/actions/info.py", line 75, in is_compatible
[2021-02-02 10:20:00.025362]     return LooseVersion(other_version) >= LooseVersion("5.0-0")
[2021-02-02 10:20:00.025386]   File "/usr/lib/python3.7/distutils/version.py", line 70, in __ge__
[2021-02-02 10:20:00.025407]     c = self._cmp(other)
[2021-02-02 10:20:00.025431]   File "/usr/lib/python3/dist-packages/univention/appcenter/app.py", line 74, in _cmp
[2021-02-02 10:20:00.025455]     for i, j in zip_longest(self.version, other.version, fillvalue=''):
[2021-02-02 10:20:00.025478] AttributeError: 'LooseVersion' object has no attribute 'version'
Comment 7 Florian Best univentionstaff 2021-02-04 10:55:11 CET
*** Bug 51326 has been marked as a duplicate of this bug. ***
Comment 8 Florian Best univentionstaff 2021-02-04 10:55:56 CET
You might want to migrate univention-app-appliance to Python 3 as well? as it depends on the appcenter:

univention-app-appliance/python/__init__.py
univention-app-appliance/usr/sbin/univention-app-appliance
Comment 9 Florian Best univentionstaff 2021-02-04 11:50:45 CET
The hashbang of univention-appcenter/scripts/univention-appcenter-listener-converter needs to be changed, if it is already Python 3 compatible.
Comment 10 Dirk Wiesenthal univentionstaff 2021-03-11 03:01:02 CET
Fixed in
  univention-appcenter 9.0.2-14A~5.0.0.202103110300
Comment 11 Dirk Wiesenthal univentionstaff 2021-03-11 03:28:14 CET
univention-app-appliance has also been migrated
Comment 12 Florian Best univentionstaff 2021-03-11 10:05:08 CET
REOPEN:

+++ base/univention-app-appliance/usr/sbin/univention-app-appliance
-               with open(dest_path) as fp:
-                       base64_img = base64.encodestring(fp.read())
+               with open(dest_path, 'rb') as fp:
+                       base64_img = base64.encodebytes(fp.read())
                portal_obj.info['background'] = base64_img

you are putting bytes into portal_obj.info which is invalid!
Also .info should never be accessed directly because it bypasses syntax validation then.

should be:
base64_img = base64.b64encode(fp.read()).decode('ASCII')
portal_obj['background'] = base64_img
Comment 13 Dirk Wiesenthal univentionstaff 2021-04-21 03:19:36 CEST
(In reply to Florian Best from comment #12)
> should be:
> base64_img = base64.b64encode(fp.read()).decode('ASCII')
> portal_obj['background'] = base64_img

Thanks, fixed.

On a side note, even with the "correct" setting of the value, UDM accepted the bytes. Syntax is Base64BaseUpload.
Comment 14 Florian Best univentionstaff 2021-04-21 14:43:17 CEST
Since UCS-5 this traceback happens: 

# univention-app
Traceback (most recent call last):
  File "/usr/bin/univention-app", line 91, in <module>
    main()
  File "/usr/bin/univention-app", line 78, in main
    ret = args.func(args)
AttributeError: 'Namespace' object has no attribute 'func'
Comment 15 Florian Best univentionstaff 2021-04-22 11:52:22 CEST
REOPEN:
use `b64encode(b'…')` without the  `replace('\n', '')`.

Traceback (most recent call last):
  File "/usr/lib/python3.7/base64.py", line 510, in _input_type_check
    m = memoryview(s)
TypeError: memoryview: a bytes-like object is required, not 'str'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/univention/management/console/base.py", line 344, in __error_handling
    six.reraise(etype, exc, etraceback)
  File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/univention/management/console/protocol/modserver.py", line 202, in _recv
    self.handle(msg)
  File "/usr/lib/python3/dist-packages/univention/management/console/protocol/modserver.py", line 270, in handle
    six.reraise(self.__init_etype, self.__init_exc, self.__init_etraceback)
  File "/usr/lib/python3/dist-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/univention/management/console/protocol/modserver.py", line 311, in handle
    self.__handler.init()
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/appcenter/__init__.py", line 157, in init
    get_action('list').get_apps()
  File "/usr/lib/python3/dist-packages/univention/appcenter/actions/list_apps.py", line 102, in get_apps
    install_permissions = app.install_permissions_exist()
  File "/usr/lib/python3/dist-packages/univention/appcenter/app.py", line 1017, in install_permissions_exist
    self._install_permissions_exist = access(image)
  File "/usr/lib/python3/dist-packages/univention/appcenter/docker.py", line 96, in access
    auth = encodestring('%s:%s' % (username, password)).replace('\n', '')
  File "/usr/lib/python3.7/base64.py", line 540, in encodestring
    return encodebytes(s)
  File "/usr/lib/python3.7/base64.py", line 527, in encodebytes
    _input_type_check(s)
  File "/usr/lib/python3.7/base64.py", line 513, in _input_type_check
    raise TypeError(msg) from err
TypeError: expected bytes-like object, not str
Comment 16 Florian Best univentionstaff 2021-04-22 11:52:42 CEST
*** Bug 53133 has been marked as a duplicate of this bug. ***
Comment 17 Felix Botner univentionstaff 2021-04-22 11:56:38 CEST
(In reply to Florian Best from comment #15)
> REOPEN:
> use `b64encode(b'…')` without the  `replace('\n', '')`.
> 
> Traceback (most recent call last):
>   File "/usr/lib/python3.7/base64.py", line 510, in _input_type_check
>     m = memoryview(s)
> TypeError: memoryview: a bytes-like object is required, not 'str'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/base.py", line
> 344, in __error_handling
>     six.reraise(etype, exc, etraceback)
>   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
>     raise value
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/protocol/
> modserver.py", line 202, in _recv
>     self.handle(msg)
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/protocol/
> modserver.py", line 270, in handle
>     six.reraise(self.__init_etype, self.__init_exc, self.__init_etraceback)
>   File "/usr/lib/python3/dist-packages/six.py", line 692, in reraise
>     raise value.with_traceback(tb)
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/protocol/
> modserver.py", line 311, in handle
>     self.__handler.init()
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/modules/
> appcenter/__init__.py", line 157, in init
>     get_action('list').get_apps()
>   File
> "/usr/lib/python3/dist-packages/univention/appcenter/actions/list_apps.py",
> line 102, in get_apps
>     install_permissions = app.install_permissions_exist()
>   File "/usr/lib/python3/dist-packages/univention/appcenter/app.py", line
> 1017, in install_permissions_exist
>     self._install_permissions_exist = access(image)
>   File "/usr/lib/python3/dist-packages/univention/appcenter/docker.py", line
> 96, in access
>     auth = encodestring('%s:%s' % (username, password)).replace('\n', '')
>   File "/usr/lib/python3.7/base64.py", line 540, in encodestring
>     return encodebytes(s)
>   File "/usr/lib/python3.7/base64.py", line 527, in encodebytes
>     _input_type_check(s)
>   File "/usr/lib/python3.7/base64.py", line 513, in _input_type_check
>     raise TypeError(msg) from err
> TypeError: expected bytes-like object, not str

please also consider 

umc/python/appcenter/__init__.py:from base64 import encodestring
umc/python/appcenter/__init__.py:						value = encodestring(value).rstrip()
Comment 18 Felix Botner univentionstaff 2021-04-22 12:27:01 CEST
here is another one

UCS 5.0, installed collabora-online, opened App Settings in UMC, new (fake) Support Key -> Apply Changes

Internal server error during "appcenter/progress".
Request: appcenter/progress

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/univention/management/console/base.py", line 344, in __error_handling
    six.reraise(etype, exc, etraceback)
  File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/univention/management/console/base.py", line 247, in execute
    function.__func__(self, request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 321, in _response
    result = _multi_response(self, request)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 181, in _response
    return function(self, request)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 443, in _response
    return list(function(self, iterator, *nones))
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 289, in _fake_func
    yield function(self, *args)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/mixins.py", line 158, in progress
    ret = progress_obj.poll()
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/mixins.py", line 102, in poll
    six.reraise(*self.exc_info)
  File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 312, in _thread
    result = _multi_response(self, request)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 181, in _response
    return function(self, request)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 443, in _response
    return list(function(self, iterator, *nones))
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 289, in _fake_func
    yield function(self, *args)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/appcenter/__init__.py", line 513, in configure
    values[setting.name] = values[setting.name].decode('base64')
AttributeError: 'str' object has no attribute 'decode'
Comment 19 Dirk Wiesenthal univentionstaff 2021-04-22 15:37:06 CEST
Fixed the encoding / decoding issues in
  univention-appcenter 9.0.2-33A~5.0.0.202104221528
Comment 20 Florian Best univentionstaff 2021-04-22 15:40:25 CEST
(In reply to Dirk Wiesenthal from comment #19)
> Fixed the encoding / decoding issues in
>   univention-appcenter 9.0.2-33A~5.0.0.202104221528
No, some of the encoding are wrong. See 3 comments in gitlab:
https://git.knut.univention.de/univention/ucs/-/commit/2c75c830b6fb84f0ebc2cd575d5df5d882ef4c77#note_17121
Comment 21 Dirk Wiesenthal univentionstaff 2021-04-22 16:10:37 CEST
Done in
  univention-appcenter 9.0.2-33A~5.0.0.202104221606
Comment 22 Felix Botner univentionstaff 2021-04-22 17:41:48 CEST
App Settings File - eg collabora-online upload a binary file in app settings (is this supposed to work?)

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/univention/management/console/base.py", line 344, in __error_handling
    six.reraise(etype, exc, etraceback)
  File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/univention/management/console/base.py", line 247, in execute
    function.__func__(self, request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 321, in _response
    result = _multi_response(self, request)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 181, in _response
    return function(self, request)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 443, in _response
    return list(function(self, iterator, *nones))
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 289, in _fake_func
    yield function(self, *args)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/mixins.py", line 158, in progress
    ret = progress_obj.poll()
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/mixins.py", line 102, in poll
    six.reraise(*self.exc_info)
  File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 312, in _thread
    result = _multi_response(self, request)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 181, in _response
    return function(self, request)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 443, in _response
    return list(function(self, iterator, *nones))
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/decorators.py", line 289, in _fake_func
    yield function(self, *args)
  File "/usr/lib/python3/dist-packages/univention/management/console/modules/appcenter/__init__.py", line 513, in configure
    values[setting.name] = b64decode(values[setting.name]).decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 1: invalid start byte
Comment 23 Dirk Wiesenthal univentionstaff 2021-04-23 10:43:48 CEST
(In reply to Felix Botner from comment #22)
> App Settings File - eg collabora-online upload a binary file in app settings
> (is this supposed to work?)
> 
> Traceback (most recent call last):
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/base.py", line
> 344, in __error_handling
>     six.reraise(etype, exc, etraceback)
>   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
>     raise value
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/base.py", line
> 247, in execute
>     function.__func__(self, request, *args, **kwargs)
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/modules/
> decorators.py", line 321, in _response
>     result = _multi_response(self, request)
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/modules/
> decorators.py", line 181, in _response
>     return function(self, request)
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/modules/
> decorators.py", line 443, in _response
>     return list(function(self, iterator, *nones))
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/modules/
> decorators.py", line 289, in _fake_func
>     yield function(self, *args)
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/modules/mixins.
> py", line 158, in progress
>     ret = progress_obj.poll()
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/modules/mixins.
> py", line 102, in poll
>     six.reraise(*self.exc_info)
>   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
>     raise value
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/modules/
> decorators.py", line 312, in _thread
>     result = _multi_response(self, request)
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/modules/
> decorators.py", line 181, in _response
>     return function(self, request)
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/modules/
> decorators.py", line 443, in _response
>     return list(function(self, iterator, *nones))
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/modules/
> decorators.py", line 289, in _fake_func
>     yield function(self, *args)
>   File
> "/usr/lib/python3/dist-packages/univention/management/console/modules/
> appcenter/__init__.py", line 513, in configure
>     values[setting.name] = b64decode(values[setting.name]).decode('utf-8')
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 1:
> invalid start byte

As discussed, I opened Bug#53138 for that. That happened to work in 4.4, but not intentionally.

Today, we have two Apps that use Type = File. And both use it for their plain text license file.
Comment 24 Florian Best univentionstaff 2021-04-23 10:49:27 CEST
(In reply to Dirk Wiesenthal from comment #23)
> 
> As discussed, I opened Bug#53138 for that. That happened to work in 4.4, but
> not intentionally.
> 
> Today, we have two Apps that use Type = File. And both use it for their
> plain text license file.

Alternatively, can you add a `setting.encoding` ?
Comment 25 Florian Best univentionstaff 2021-04-26 13:25:49 CEST
REOPEN:
management/univention-appcenter/python/appcenter/udm.py
206 »   »   »   »   »   base64icon = base64.b64encode(f.read())
→ wrong, as it returns a bytestring.

If possible please provide a test case.
Comment 26 Florian Best univentionstaff 2021-04-26 13:26:51 CEST
  1 ./python/appcenter/packages.py|173 col 20 warning| invalid escape sequence '\(' [W605]                                                                                                                                                     
  2 ./python/appcenter/packages.py|174 col 25 warning| invalid escape sequence '\[' [W605]
  3 ./python/appcenter/packages.py|174 col 32 warning| invalid escape sequence '\]' [W605]
Comment 27 Florian Best univentionstaff 2021-04-26 16:20:22 CEST
management/univention-appcenter/python/appcenter/extended_attributes.py
  190 »   »   return 'cn=%s,%s,%s' % (self.name, self.position, ucr_get('ldap/base'))
  194 »   »   kwargs.setdefault('position', 'cn=%s,cn=custom attributes,cn=univention' % app.id)
  241 »   »   kwargs.setdefault('position', 'cn=%s,cn=custom attributes,cn=univention' % (app.id,))
  252 »   »   return 'cn=%s,%s,%s' % (self.name, self.position, ucr_get('ldap/base'))

→ Please fix LDAP DN char escaping.

management/univention-appcenter/python/appcenter/actions/register.py
286 »   »   »   ldap_filter = '(|%s)' % ''.join('(univentionObjectType=%s)' % udm_module for udm_module in app.listener_udm_modules)
→ Please fix LDAP filer escaping

297 name = '%(name)s'
301 »   »   name = '%(name)s'
305 »   »   ldap_filter = '%(ldap_filter)s'
306 »   »   dump_dir = '%(dump_dir)s'
307 »   »   output_dir = '%(output_dir)s'
→ Please use %r representations, so that strings are correctly escaped.

management/univention-appcenter/python/appcenter/database.py
252 »   »   stdout = self.execute('SELECT COUNT(*) FROM pg_database WHERE datname = \'%s\'' % self.get_db_name())
261 »   »   stdout = self.execute('SELECT COUNT(*) FROM pg_user WHERE usename = \'%s\'' % self.get_db_user())
→ fix SQL escaping

management/univention-appcenter/python/appcenter/utils.py
  356 def get_sha256_from_file(filename):
  357 »   if os.path.exists(filename):
  358 »   »   with open(filename, 'r') as f:
  359 »   »   »   return get_sha256(f.read())

→ read file as binary, otherwise UTF-8 will be used, which breaks for binary data. Also backwards compatibility with the resulting sha hashes is broken.


management/univention-appcenter/python/appcenter/utils.py
   581 »   »   »   if lo.search('(&(univentionObjectType=appcenter/app)(univentionAppInstalledOnServer=*)(univentionAppID=%s_*))' % required_app.id):

→ fix LDAP filter escaping

ucs/management/univention-appcenter/umc/python/appcenter/__init__.py
   580 »   »   »   »   »   values[setting.name] = values[setting.name].decode('base64')
→ fix base64 decoding
Comment 28 Dirk Wiesenthal univentionstaff 2021-04-30 02:36:42 CEST
[5.0-0 2dc7f768ca] Bug #51598: QA Feedback
 5 files changed, 15 insertions(+), 20 deletions(-)


(In reply to Florian Best from comment #25)
> REOPEN:
> management/univention-appcenter/python/appcenter/udm.py
> 206 »   »   »   »   »   base64icon = base64.b64encode(f.read())
> → wrong, as it returns a bytestring.
> 
> If possible please provide a test case.

Removed that part. Relevant appcenter/app objects' attributes are just ID, version, servers. If this changes, we would need a new approach anyway.

(In reply to Florian Best from comment #26)
>   1 ./python/appcenter/packages.py|173 col 20 warning| invalid escape
> sequence '\(' [W605]                                                        
> 
>   2 ./python/appcenter/packages.py|174 col 25 warning| invalid escape
> sequence '\[' [W605]
>   3 ./python/appcenter/packages.py|174 col 32 warning| invalid escape
> sequence '\]' [W605]

Done

(In reply to Florian Best from comment #27)
> management/univention-appcenter/python/appcenter/extended_attributes.py
>   190 »   »   return 'cn=%s,%s,%s' % (self.name, self.position,
> ucr_get('ldap/base'))
>   194 »   »   kwargs.setdefault('position', 'cn=%s,cn=custom
> attributes,cn=univention' % app.id)
>   241 »   »   kwargs.setdefault('position', 'cn=%s,cn=custom
> attributes,cn=univention' % (app.id,))
>   252 »   »   return 'cn=%s,%s,%s' % (self.name, self.position,
> ucr_get('ldap/base'))
> 
> → Please fix LDAP DN char escaping.

Done

> 297 name = '%(name)s'
> 301 »   »   name = '%(name)s'
> 305 »   »   ldap_filter = '%(ldap_filter)s'
> 306 »   »   dump_dir = '%(dump_dir)s'
> 307 »   »   output_dir = '%(output_dir)s'
> → Please use %r representations, so that strings are correctly escaped.

Done

> management/univention-appcenter/python/appcenter/utils.py
>   356 def get_sha256_from_file(filename):
>   357 »   if os.path.exists(filename):
>   358 »   »   with open(filename, 'r') as f:
>   359 »   »   »   return get_sha256(f.read())
> 
> → read file as binary, otherwise UTF-8 will be used, which breaks for binary
> data. Also backwards compatibility with the resulting sha hashes is broken.
> 

Done
Comment 29 Dirk Wiesenthal univentionstaff 2021-05-05 22:17:01 CEST
commit d809675c96c886de7a5ab9d1622e48679e07b5e6 (HEAD -> 5.0-0, origin/5.0-0)
Author: Dirk Wiesenthal <wiesenthal@univention.de>

    Bug #51598: Improve SQL query building

commit 302d2078fd7aa54c835710935480eaf643f84227
Author: Dirk Wiesenthal <wiesenthal@univention.de>

    Bug #51598: Remove outdated UMC commands


(In reply to Florian Best from comment #27)
> management/univention-appcenter/python/appcenter/actions/register.py
> 286 »   »   »   ldap_filter = '(|%s)' % ''.join('(univentionObjectType=%s)'
> % udm_module for udm_module in app.listener_udm_modules)
> → Please fix LDAP filer escaping
> 

Done

> 
> management/univention-appcenter/python/appcenter/database.py
> 252 »   »   stdout = self.execute('SELECT COUNT(*) FROM pg_database WHERE
> datname = \'%s\'' % self.get_db_name())
> 261 »   »   stdout = self.execute('SELECT COUNT(*) FROM pg_user WHERE
> usename = \'%s\'' % self.get_db_user())
> → fix SQL escaping
> 

Done

> management/univention-appcenter/python/appcenter/utils.py
>    581 »   »   »   if
> lo.search('(&(univentionObjectType=appcenter/
> app)(univentionAppInstalledOnServer=*)(univentionAppID=%s_*))' %
> required_app.id):
> 
> → fix LDAP filter escaping
> 

Done

> ucs/management/univention-appcenter/umc/python/appcenter/__init__.py
>    580 »   »   »   »   »   values[setting.name] =
> values[setting.name].decode('base64')
> → fix base64 decoding

Done by removing this code. The UMC method was not used anyway.
Comment 30 Dirk Wiesenthal univentionstaff 2021-05-05 22:57:54 CEST
And

[5.0-0 c2253d5ef7] Bug #51598: Fix issues found by linting
 9 files changed, 19 insertions(+), 15 deletions(-)
Comment 31 Florian Best univentionstaff 2021-05-05 23:04:31 CEST
OK: I did a code review around every single line, the obvious things are covered
OK: jenkins tests
OK: changelog entry
Comment 32 Florian Best univentionstaff 2021-05-25 16:03:29 CEST
UCS 5.0 has been released:
 https://docs.software-univention.de/release-notes-5.0-0-en.html
 https://docs.software-univention.de/release-notes-5.0-0-de.html

If this error occurs again, please use "Clone This Bug".
Comment 33 Iván.Delgado univentionstaff 2023-06-27 15:34:53 CEST
ff770e527713 | Bug #51598: Fix syntax error appliance prepare_docker_app
31ab7ec742f9 | Bug #51598: Fix syntax error appliance prepare_docker_app