Univention Bugzilla – Bug 45024
[RESTful Import API] server implementation
Last modified: 2021-06-03 17:01:28 CEST
Create a server implementation of the RESTful API that allows a UCS@school customer to remotely do UCS@school user imports.
* Access to the API will be restricted to users in groups as configured through Bug #45026. A member of a group will be able to start imports and read results for all schools listed in that group. * A configuration file per school will be loaded with filename <ou>.json. If the file cannot be found, a default (fallback) will be used.
r81381: First WIP(!) commit of ucs-school-import-http-api package. After installing you'll find a self-documenting API at https://<FQDN/IP>/api/v1/ File locations in the Debian package were built as close to https://wiki.debian.org/DjangoPackagingDraft as possible. There is no finished standard for packaging Django apps. The join script is designed so that it can be run as often as desired, fixing as much as possible. If you wish to "start over" with your data, the fastest way is to simply delete the database and recreate it: # su - postgres -c "dropdb importhttpapi" # univention-run-join-scripts --force --run-scripts 40ucs-school-import-http-api.inst * The /schools/ endpoint is functional. * /schools/{ou}/imports/users (or /imports/users) allows to POST data into it, and list it, but no import job is started yet.
PS: Authentication is done through PAM.
r81566: add endpoints for logfile, password and summary subresources r81577: added support for user type DB schema changed, please run: $ su - postgres -c "dropdb importhttpapi" $ univention-run-join-scripts --force --run-scripts 40ucs-school-import-http-api.inst Worker code changed, please run: $ systemctl restart celery-worker-ucsschool-import.service
r81581: add 'name' property to School resource r81677: block installation of Python 3 version of Celery r81779: add support for paging, filtering and ordering r81803,81804: * add support for school/role restrictions * remove unneeded features * remove per job hook support * remove per job configuration file support * reduce models * handle existing super administrator in join script r81806: remove bashism ucs-school-import (15.0.0-24)
r81848: make school, source_uid and user_role optional r81869: check text atrifact access permission, load text artifact content in admin view r81875: add progress update support r81876 (ucs-school-lib): add support for changing path of hooks r81877: require school lib with support for changeable hook directory r81878: advisories Package: ucs-school-lib Version: 10.0.2-4A~4.2.0.201708080738 Package: ucs-school-import Version: 15.0.0-32A~4.2.0.201708080740 The progress of the import job is now written to UserImportJob.result.result which is a dict: {'total': 0, 'done': 0, 'stage': '1/4 initializing'} {'total': 0, 'done': 0, 'stage': '2/4 analyzing data'} {'total': 3, 'done': 3, 'stage': '3/4 deleting users'} {'total': 132, 'done': 4, 'stage': '4/4 creating / modifying users'} {'total': 132, 'done': 54, 'stage': '4/4 creating / modifying users'} When finished, it is set to the return value of the celery task: 'UserImportJob #34 ended successfully.'
r81901 (ucs-school-import 15.0.0-33): * UserImportJob.result.result is always dict * result has percentage in str and int * remove duplicate status attribute {"percentage": 0, "total": 0, "done": 0, "description": "Initializing: 0%."} {"percentage": 26, "total": 100, "done": 18, "errors": 0, "description": "Creating and modifying users: 26%."} {"percentage": 100, "total": 0, "done": 0, "description": "UserImportJob #10 ended successfully."}
ucs-school-import-http-api (15.0.0-33A~4.2.0.201708081637) wird eingerichtet ... File: /etc/apache2/sites-available/ucs-school-import-http-api.conf File: /etc/default/celery-worker-ucsschool-import File: /etc/ucsschool-import/settings.py File: /etc/init.d/celery-worker-ucsschool-import File: /etc/gunicorn.d/ucs-school-import Calling joinscript 40ucs-school-import-http-api.inst ... 2017-08-16 15:55:44.258369253+02:00 (in joinscript_init) Object created: cn=ucsschoolGroup,cn=UCSschool,cn=custom attributes,cn=univention,dc=univention,dc=intranet Object created: cn=ucsschoolSchool-ImportSchool,cn=UCSschool,cn=custom attributes,cn=univention,dc=univention,dc=intranet Object created: cn=ucsschoolSchool-ImportRole,cn=UCSschool,cn=custom attributes,cn=univention,dc=univention,dc=intranet Creating secret for HTTP API service. Creating new password for RabbitMQ. Creating new RabbitMQ user and vhost. Creating user "importhttpapi" ... ...done. Creating vhost "importhttpapi" ... ...done. Setting permissions for user "importhttpapi" in vhost "importhttpapi" ... ...done. Creating new password for PostgreSQL. Creating new PostgreSQL user. CREATE ROLE ALTER ROLE Creating new PostgreSQL database. Traceback (most recent call last): File "/usr/share/pyshared/ucsschool/http_api/manage.py", line 23, in <module> execute_from_command_line(sys.argv) File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line utility.execute() File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 328, in execute django.setup() File "/usr/lib/python2.7/dist-packages/django/__init__.py", line 18, in setup apps.populate(settings.INSTALLED_APPS) File "/usr/lib/python2.7/dist-packages/django/apps/registry.py", line 85, in populate app_config = AppConfig.create(entry) File "/usr/lib/python2.7/dist-packages/django/apps/config.py", line 86, in create module = import_module(entry) File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/usr/lib/python2.7/dist-packages/django_filters/__init__.py", line 3, in <module> from .filterset import FilterSet File "/usr/lib/python2.7/dist-packages/django_filters/filterset.py", line 10, in <module> from django.db.models.related import RelatedObject ImportError: No module named related Joinscript 40ucs-school-import-http-api.inst finished with exitcode 1 This is caused by python-django-filter and python-django-filter which are simultaneously installed. Added a conflict to "python-django-filters (<<1.0.0)" for ucs-school-import-http-api. ucs-school-import (15.0.0-34): r82595 | Bug #45024: fixed dependencies
Restrict the list of schools in the API to those the logged in user has permissions to start imports on.
(In reply to Daniel Tröder from comment #9) > Restrict the list of schools in the API to those the logged in user has > permissions to start imports on. Moved to Bug #45044.
Why has settings.py been implemented as UCR template? There are no python sections or keyword replacements evaluated by UCR. A much simpler solution would be the installation of the file via debian/ucs-school-import-http-api.install to /etc/ucsschool-import/settings.py → REOPEN In 65e887fcc02706ddede9291025b9f7d077ac0277 objectclasses for students, teachers, staff, ... are hardcoded in ucs-school-import. I would prefer, that the dictionary is "somehow" moved to ucs-school-lib/python/models/user.py into the class User(UCSSchoolHelperAbstractClass). This way, there is a single point of configuration. The "User" class itself should also reuse the new objectclass definition in its own methods. → REOPEN
(In reply to Sönke Schwardt-Krummrich from comment #11) > Why has settings.py been implemented as UCR template? There are no python > sections or keyword replacements evaluated by UCR. > A much simpler solution would be the installation of the file via > debian/ucs-school-import-http-api.install to > /etc/ucsschool-import/settings.py > → REOPEN This was resolved in commits 574b882..ad1dd8a * don't manage settings file with UCR * add UCRV descriptions * get time zone from system > In 65e887fcc02706ddede9291025b9f7d077ac0277 objectclasses for students, > teachers, staff, ... are hardcoded in ucs-school-import. I would prefer, > that the dictionary is "somehow" moved to > ucs-school-lib/python/models/user.py into > the class User(UCSSchoolHelperAbstractClass). This way, there is a single > point of configuration. The "User" class itself should also reuse the new > objectclass definition in its own methods. > → REOPEN This was resolved in commits 4fe802a..9c14bf2 The Bug #45044 was used in commits, advisories and changelogs. Package: ucs-school-lib Version: 10.0.2-6A~4.2.0.201709141720 Branch: ucs_4.2-0 Scope: ucs-school-4.2 Package: ucs-school-import Version: 15.0.0-41A~4.2.0.201709141722 Branch: ucs_4.2-0 Scope: ucs-school-4.2 advisories: 6185d506cb21f9f1817390d88400acefd45bfcda
Move /var/spool/ucs-school-import/jobs/ to /var/lib/ucs-school-import/jobs/
Job data is now stored in /var/lib/ucs-school-import/jobs/$YEAR/$JOBID/. [4.2 88d39297] Bug #45024: move import job directory to /var/lib [4.2 9ea2cc50] Bug #45024: update advisory Package: ucs-school-import Version: 15.0.3-7A~4.2.0.201711201503
OK: the new directory is used.
UCS@school 4.2 v6 has been released. http://docs.software-univention.de/changelog-ucsschool-4.2v6-de.html If this error occurs again, please clone this bug.
*** Bug 41241 has been marked as a duplicate of this bug. ***