Bug 53599 - Userimport not usable when UCS@school has been installed while bug 53576 was not fixed
Userimport not usable when UCS@school has been installed while bug 53576 was ...
Status: NEEDMOREINFO
Product: UCS@school
Classification: Unclassified
Component: UMC - User Import UI
UCS@school 4.4
Other Linux
: P5 normal (vote)
: ---
Assigned To: UCS@school maintainers
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2021-07-20 11:42 CEST by Christina Scheinig
Modified: 2021-08-20 15:39 CEST (History)
6 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 1: Will affect a very few installed domains
How will those affected feel about the bug?: 2: A Pain – users won’t like this once they notice it
User Pain: 0.057
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2021071221000289, 2021071921000427, 2021072021000684, 2021071921000356, 2021081721000232, 2021080321000384
Bug group (optional): Workaround is available
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Christina Scheinig univentionstaff 2021-07-20 11:42:23 CEST
Opening the User Import Module throws the following traceback:


An error occurred
Could not process the request.
Internal server error during "schoolimport/schools".

An error occurred
Internal server error.

Server error message:

The UCS@school Import API HTTP server is not reachable: Received status_code=500 with reason='Internal Server Error' for requests.get(**files=None, url=u'https://school.schein.en/api/v1/imports/users/', auth=(u'Administrator', u'**********'), headers={u'Accept': u'application/json'}, params={'ordering': 'date_created', 'limit': 20, 'dryrun': False}, data=None).


20.07.21 11:39:13.416  MAIN        ( ERROR   ) : Internal server error during "schoolimport/schools".
Request: schoolimport/schools

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/univention/management/console/base.py", line 359, in __error_handling
    six.reraise(etype, exc, etraceback)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/base.py", line 262, in execute
    function.__func__(self, request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/modules/decorators.py", line 641, in _decorated
    return function(self, request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/modules/decorators.py", line 321, in _response
    result = _multi_response(self, request)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/modules/decorators.py", line 181, in _response
    return function(self, request)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/modules/decorators.py", line 443, in _response
    return list(function(self, iterator, *nones))
  File "/usr/lib/python2.7/dist-packages/univention/management/console/modules/decorators.py", line 289, in _fake_func
    yield function(self, *args)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/modules/schoolimport/__init__.py", line 104, in schools
    dict(id=school.name, label=school.displayName) for school in self.client.school.list()
  File "/usr/lib/python2.7/dist-packages/ucsschool/http_api/client.py", line 593, in list
    return self._to_python(self._list_resource(**params))
  File "/usr/lib/python2.7/dist-packages/ucsschool/http_api/client.py", line 551, in _list_resource
    return self._resource_from_url(resource_url, **params)
  File "/usr/lib/python2.7/dist-packages/ucsschool/http_api/client.py", line 543, in _resource_from_url
    return self.client.call_api("get", url, params=params)
  File "/usr/lib/python2.7/dist-packages/ucsschool/http_api/client.py", line 524, in call_api
    raise exc(msg, status_code=response.status_code)
ServerError: Received status_code=500 with reason='Internal Server Error' for requests.get(**files=None, url=u'https://school.schein.en/api/v1/schools/', auth=(u'Administrator', u'**********'), headers={u'Accept': u'application/json'}, params={}, data=None).
20.07.21 11:39:13.558  MAIN        ( PROCESS ) : CPCommand (10.205.2.18:59316) response status code: 500
20.07.21 11:39:13.558  MAIN        ( PROCESS ) : CPCommand (10.205.2.18:59316) response reason : None
20.07.21 11:39:13.558  MAIN        ( PROCESS ) : CPCommand (10.205.2.18:59316) response message: The UCS@school Import API HTTP server is not reachable: Received status_code=500 with reason='Internal Server Error' for requests.get(**files=None, url=u'https://school.schein.en/api/v1/imports/users/', auth=(u'Administrator', u'**********'), headers={u'Accept': u'application/json'}, params={'ordering': 'date_created', 'limit': 20, 'dryrun': False}, data=None).
root@school:~# dpkg -l |grep ucs-school-import-http-api
ii  ucs-school-import-http-api                          17.0.71A~4.4.0.202107141220                                         all          UCS@school: RESTful API for importing users
ii  ucs-school-import-http-api-client                   17.0.71A~4.4.0.202107141220                                         all          UCS@school: Python client library for the US@school API
ii  ucs-school-import-http-api-common                   17.0.71A~4.4.0.202107141220                                         all          UCS@school: Common files for UCS@school API client and server
------------------------
root@school:~# dpkg -l |grep python-ucsschool-importer-http-api
ii  python-ucsschool-importer-http-api                  17.0.71A~4.4.0.202107141220                                         all          UCS@school: RESTful API for importing users
ii  python-ucsschool-importer-http-api-client           17.0.71A~4.4.0.202107141220                                         all          UCS@school: Python client library for the US@school API
ii  python-ucsschool-importer-http-api-common           17.0.71A~4.4.0.202107141220                                         all          UCS@school: Common files for UCS@school API client and server

-----------------------------------------------------------------------
root@school:~# ps aufx |grep gunico
root       971  0.0  0.5  67036 22664 ?        Ss   11:32   0:00 gunicorn: master [ucsschool.http_api.app.wsgi:application]
uas-imp+  1668  0.2  3.3 708716 136252 ?       S    11:32   0:01  \_ gunicorn: worker [ucsschool.http_api.app.wsgi:application]
uas-imp+  1671  0.2  3.3 708984 136360 ?       S    11:32   0:01  \_ gunicorn: worker [ucsschool.http_api.app.wsgi:application]
root      3540  0.0  0.0  14316   964 pts/0    S+   11:41   0:00          \_ grep gunico
root@school:~# ps aufx |grep celery
root      3544  0.0  0.0  14316   968 pts/0    S+   11:41   0:00          \_ grep celery
root      2439  0.3  2.9 613616 118912 ?       S    11:33   0:01 [celeryd: dryrun@school:MainProcess] -active- (worker --app=ucsschool.http_api.app.celery:app --loglevel=INFO -l DEBUG -O fair -Q dryrun -c 4 --logfile=/var/log/univention/ucs-school-import/workers-dryrun.log --pidfile=/run/ucs-school-import-worker/dryrun.pid --hostname=dryrun@school)
root      2914  0.0  2.3 613360 95896 ?        S    11:33   0:00  \_ [celeryd: dryrun@school:Worker-1]
root      2916  0.0  2.3 613360 95904 ?        S    11:33   0:00  \_ [celeryd: dryrun@school:Worker-2]
root      2917  0.0  2.3 613360 95908 ?        S    11:33   0:00  \_ [celeryd: dryrun@school:Worker-3]
root      2919  0.0  2.3 613360 95908 ?        S    11:33   0:00  \_ [celeryd: dryrun@school:Worker-4]
root      2762  0.2  2.9 613616 119092 ?       S    11:33   0:01 [celeryd: import@school:MainProcess] -active- (worker --app=ucsschool.http_api.app.celery:app --loglevel=INFO -l DEBUG -O fair -Q import -c 1 --logfile=/var/log/univention/ucs-school-import/workers-import.log --pidfile=/run/ucs-school-import-worker/import.pid --hostname=import@school)
root      2924  0.0  2.3 613360 96016 ?        S    11:33   0:00  \_ [celeryd: import@school:Worker-1]
root@school:~# ps aufx |grep rabbit
rabbitmq   980  0.0  0.0   4276   696 ?        Ss   11:32   0:00 /bin/sh /usr/sbin/rabbitmq-server
rabbitmq  1000  0.0  0.0   4276  1428 ?        S    11:32   0:00  \_ /bin/sh -e /usr/lib/rabbitmq/bin/rabbitmq-server
rabbitmq  1799  0.2  1.1 1670404 45648 ?       Sl   11:33   0:01      \_ /usr/lib/erlang/erts-8.2.1/bin/beam -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 32000 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-0.0.0/ebin -noshell -noinput -s rabbit boot -sname rabbit@school -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@school.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@school-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-0.0.0/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@school-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@school" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
rabbitmq  2096  0.0  0.0   4172  1336 ?        Ss   11:33   0:00          \_ erl_child_setup 65536
rabbitmq  2749  0.0  0.0  11420   992 ?        Ss   11:33   0:00              \_ inet_gethost 4
rabbitmq  2750  0.0  0.0  11420   108 ?        S    11:33   0:00                  \_ inet_gethost 4
root      3549  0.0  0.0  14316   968 pts/0    S+   11:41   0:00          \_ grep rabbit
rabbitmq  1369  0.0  0.0  30540   244 ?        S    11:32   0:00 /usr/lib/erlang/erts-8.2.1/bin/epmd -daemon
-------------------------------------------------------------------


Still not working with the fix of  Bug 53576
Comment 1 Florian Best univentionstaff 2021-07-20 21:05:52 CEST
Did you reinstall your setup with fixed Bug #53576?
Comment 2 Tobias Wenzel univentionstaff 2021-07-21 10:29:02 CEST
Yes I debugged the system yesterday. 

univention-app info
UCS: 4.4-8 errata1009
Installed: cups=2.2.1 samba4=4.10 squid=3.5 ucsschool=4.4 v9 ucsschool-veyon-proxy=1.1

The package version is 17.0.71

Testing the http-api like

python /usr/share/pyshared/ucsschool/http_api/manage.py shell -c \
    "from ucsschool.http_api.import_api.models import School; print School.objects.all()"

does not work any more due to the dh-python migration. We have to adjust the path in our documentation for that:

python /usr/lib/python2.7/dist-packages/ucsschool/http_api/manage.py shell -c \
    "from ucsschool.http_api.import_api.models import School; print School.objects.all()"

This raised a traceback which should have been fixed with Bug #53576?


Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ucsschool/http_api/manage.py", line 24, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/dist-packages/django/core/management/commands/shell.py", line 114, in handle
    exec(options['command'])
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 232, in __repr__
    data = list(self[:REPR_OUTPUT_SIZE + 1])
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 256, in __iter__
    self._fetch_all()
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 1087, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 54, in __iter__
    results = compiler.execute_sql()
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python2.7/dist-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: FEHLER:  Relation »import_api_school« existiert nicht
LINE 1: ...l"."name", "import_api_school"."displayName" FROM "import_ap...


The joinscript ran without failing, but when I executed them again, the error was not raised. 

univention-run-join-scripts --run-scripts 40ucs-school-import-http-api.inst --force

There might be something wrong with the execution order in the joinscript. This has to be fixed.
Comment 3 Erik Damrose univentionstaff 2021-07-21 12:52:37 CEST
As far as the issue is understood, this occurs when UCS@school has been installed while bug 53576 was not yet fixed. Re-running the http-api joinscript (40ucs-school-import-http-api.inst) fixes the problem.

https://help.univention.com/t/18288
Comment 4 Christina Scheinig univentionstaff 2021-07-22 09:17:56 CEST
Exactly.

But, the bug was created, before the workaround was known.
So I don't know if there is still something more to do. Maybe checking installation order or checking the join during installation, because something has not yet been quite right here. And this could cause problems next time?
But this is not for me to decide.
Comment 5 Christina Scheinig univentionstaff 2021-07-26 13:36:47 CEST
We have this issue still in the forum, and the workaround does not solve the problem.
Comment 6 Daniel Tröder univentionstaff 2021-08-02 08:51:04 CEST
(In reply to Erik Damrose from comment #3)
> As far as the issue is understood, this occurs when UCS@school has been
> installed while bug 53576 was not yet fixed. Re-running the http-api
> joinscript (40ucs-school-import-http-api.inst) fixes the problem.
> 
> https://help.univention.com/t/18288

Doesn't that mean, that the fix for bug 53576 should have raised the join script VERSION?
Comment 7 Erik Damrose univentionstaff 2021-08-02 09:17:14 CEST
(In reply to Daniel Tröder from comment #6)
> Doesn't that mean, that the fix for bug 53576 should have raised the join
> script VERSION?

Thats currently unclear. Right now we think that bug 53576 caused the joinscript action to initialize the DB failed silently, so re-running it with the fixed package fixes the issue.