Univention Bugzilla – Bug 53451
Replace univention-celery with a systemd configuration for UCS@school
Last modified: 2021-11-29 17:20:21 CET
In UCS 5.0 univention-celery has been removed. The ucs-school-import uses a univention-celery configuration. We need to implement a systemd replacement.
Currently the ucs-school-import-http-api cannot be installed. It's only a Recommends in UCS@school. # LANG=C apt install ucs-school-import-http-api Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: ucs-school-import-http-api : Depends: univention-celery (>= 1.0.0-1) but it is not installable E: Unable to correct problems, you have held broken packages.
Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/dist-packages/ucsschool/http_api/manage.py", line 24, in <module> execute_from_command_line(sys.argv) File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 364, in execute_from_command_line utility.execute() File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 338, in execute django.setup() File "/usr/lib/python2.7/dist-packages/django/__init__.py", line 27, in setup apps.populate(settings.INSTALLED_APPS) File "/usr/lib/python2.7/dist-packages/django/apps/registry.py", line 108, in populate app_config.import_models() File "/usr/lib/python2.7/dist-packages/django/apps/config.py", line 202, in import_models self.models_module = import_module(models_module_name) File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/usr/lib/python2.7/dist-packages/djcelery/models.py", line 15, in <module> from celery.utils.timeutils import timedelta_seconds ImportError: No module named timeutils
(In reply to Florian Best from comment #2) > Traceback (most recent call last): > File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main > "__main__", fname, loader, pkg_name) > File "/usr/lib/python2.7/runpy.py", line 72, in _run_code > exec code in run_globals > File "/usr/lib/python2.7/dist-packages/ucsschool/http_api/manage.py", line > 24, in <module> > execute_from_command_line(sys.argv) > File > "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line > 364, in execute_from_command_line > utility.execute() > File > "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line > 338, in execute > django.setup() > File "/usr/lib/python2.7/dist-packages/django/__init__.py", line 27, in > setup > apps.populate(settings.INSTALLED_APPS) > File "/usr/lib/python2.7/dist-packages/django/apps/registry.py", line 108, > in populate > app_config.import_models() > File "/usr/lib/python2.7/dist-packages/django/apps/config.py", line 202, > in import_models > self.models_module = import_module(models_module_name) > File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module > __import__(name) > File "/usr/lib/python2.7/dist-packages/djcelery/models.py", line 15, in > <module> > from celery.utils.timeutils import timedelta_seconds > ImportError: No module named timeutils The problem here is that the django-celery version is incompatible with the celery version. As a workaround I am trying to include the UCS 4.3 repositories: echo 'deb https://updates.software-univention.de/4.3/maintained/ 4.3-0/all/' > /etc/apt/sources.list.d/21_ucsschool_workaround And make dependencies on: python-celery (= 3.1.23-7), python-celery-common (= 3.1.23-7), python-kombu (= 3.0.35-1.1), With this I am able to run the celery service under UCS@school 5.0 again. Let's hope, that the Jenkins jobs do as well.
django-celery isn't compatible with Django 1.11. So we cannot use it. Using UCS 4.3 will not be Python 3 compatible. AFAICS we don't need django-celery, we do need django-celery-results though: https://docs.celeryproject.org/en/latest/django/first-steps-with-django.html But django-celery-results exists only for Python 3. So we are at a crossroads here… I'll talk to the PO.
(In reply to Daniel Tröder from comment #4) > django-celery isn't compatible with Django 1.11. So we cannot use it. OK! > Using UCS 4.3 will not be Python 3 compatible. It would also be no real option, it's just a workaround for now so that the tests pass. > AFAICS we don't need django-celery, we do need django-celery-results though: > https://docs.celeryproject.org/en/latest/django/first-steps-with-django.html Ok, that's great! It's available in debian buster: https://packages.debian.org/buster/python3-django-celery-results > But django-celery-results exists only for Python 3. > So we are at a crossroads here… I'll talk to the PO. That's no problem. UCS@school 5.0 will mostly be Python 3 only.
I guess the following traceback is currently also caused by the workaround. Traceback (most recent call last): File "01_test_school_creation_http_api_update.py", line 29, in <module> test_school_creation() File "01_test_school_creation_http_api_update.py", line 25, in test_school_creation assert name in list(School.objects.all().values_list("name", flat=True)) File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 250, in __iter__ self._fetch_all() File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 1121, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 152, in __iter__ for row in compiler.results_iter(chunked_fetch=self.chunked_fetch): File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 841, in results_iter results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch) File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 899, in execute_sql raise original_exception django.db.utils.ProgrammingError: FEHLER: Relation »import_api_school« existiert nicht LINE 1: SELECT "import_api_school"."name" FROM "import_api_school" O...
I created an example migration / starting point (untested yet): https://git.knut.univention.de/univention/ucsschool/-/commit/ab21341c0ae779b71be39e77c7a6983d7be8026b
(In reply to Florian Best from comment #5) > (In reply to Daniel Tröder from comment #4) > > So we are at a crossroads here… I'll talk to the PO. > That's no problem. UCS@school 5.0 will mostly be Python 3 only. That's new to me - and our customers that have import hooks.
(In reply to Daniel Tröder from comment #8) > (In reply to Florian Best from comment #5) > > (In reply to Daniel Tröder from comment #4) > > > So we are at a crossroads here… I'll talk to the PO. > > That's no problem. UCS@school 5.0 will mostly be Python 3 only. > That's new to me - and our customers that have import hooks. The UCS@school import service must run with one Python version. We can and should not make this configurable per System. The minimum requirement for customers is that their Python hooks are Python 3 compatible. It would not make sense to offer a Python 2 service here.
We talked about this with the PO and decided, that having a Python3-only service is what we want :)
Let's keep the scope of this bug small. The celery configuration from UCS has been cherry-picked into UCS@school. Then the init script has been migrated to a systemd service unit. djcelery has been replaced with django-celery-results. celery and django now run with Python 3. gunicorn now runs with Python 3. (In reply to Florian Best from comment #6) > I guess the following traceback is currently also caused by the workaround. > > django.db.utils.ProgrammingError: FEHLER: Relation »import_api_school« > existiert nicht > LINE 1: SELECT "import_api_school"."name" FROM "import_api_school" O... This error drived me crazy, but it was caused by Bug #53576 and has nothing to do with this bug here. All services are running. If something does not yet work completely than it's in the scope of Bug #52609. ucs-test-ucsschool (7.0.2) 68e1afcd3d75 | Bug #53451: replace djcelery with django-celery-results ucs-school-import (18.0.1) ea79e9719375 | Bug #53451: migrate celery from init.d to systemd afc1831ea6ba | Bug #53451: run gunicorn with python 3 68e1afcd3d75 | Bug #53451: replace djcelery with django-celery-results 51d378a27859 | Bug #53451: (re)introduce ucs-school-celery ucs-school-import (17.0.70) 76e84582cb2d | Bug #53451: make files findable via git grep
In the daemon.log I can see this when trying to use the ucsschool-umc-module: Aug 26 09:17:48 master203 sh[32204]: #011> import@master203: /usr/bin/python3 -m celery worker --detach --pidfile=/run/ucs-school-import-worker/import.pid --logfile=/var/log/univention/ucs-school-import/workers-import.log -A ucsschool.http_api.app.celery:app -c 1 -Q import -O fair -l DEBUG -n import@master203 --executable=/usr/bin/python3 Aug 26 09:17:48 master203 sh[32204]: #033[1;32mOK#033[0m Aug 26 09:17:48 master203 sh[32204]: Traceback (most recent call last): Aug 26 09:17:48 master203 sh[32204]: File "/usr/lib/python3/dist-packages/celery/platforms.py", line 148, in acquire Aug 26 09:17:48 master203 sh[32204]: self.write_pid() Aug 26 09:17:48 master203 sh[32204]: File "/usr/lib/python3/dist-packages/celery/platforms.py", line 211, in write_pid Aug 26 09:17:48 master203 sh[32204]: pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE) Aug 26 09:17:48 master203 sh[32204]: FileNotFoundError: [Errno 2] No such file or directory: '/run/ucs-school-import-worker/import.pid' The directory was created by the init script which does not exist yet. We should add something like: ExecStartPre=-/usr/bin/mkdir /run/ucs-school-import-worker to the systemd service file
The systemd unit now creates the directories as before. The service name is hard to remember, so I renamed it to ucs-school-import-celery-worker and put an alias for the old name. ucs-school-import (18.0.1) 40263eda0d6a | Bug #53451: rename celery-worker-ucsschool-import.service into ucs-school-import-celery-worker.service ed4435e4e47e | fixup! Bug #53451: migrate celery from init.d to systemd
Rename: OK Creation of PID-File dir with ExecStartPre:OK Service starting and running: OK Graphical import with incorrect file: OK, Error is shown Graphical import with correct import file: OK works Updated changlog entry: OK Verified
UCS@school 5.0 v1 has been released. https://docs.software-univention.de/release-notes-ucsschool-5.0v1-de.html If this error occurs again, please clone this bug.