Univention Bugzilla – Bug 53383
UMC Import Module is slow with many schools
Last modified: 2022-08-29 16:45:20 CEST
When I login as a "global school administrator" (-> who has import permissions for all schools) and want to import new users via the UCS@School-Module "user import" it takes too long to be usable. Example from a productive environment: * 240 successful imports * 300 schools * about 50k users -> wait time for the module to load: 45s This is reproducible on our test environment. Guess: the module has no view limitation and tries to load all schools and import jobs, which takes very long in bigger environments
How relevant is this scenario? The UMC-Import was created for a distributed import scenario, where people at different schools do imports. Each of those people will only see one or a few schools.
This may be a duplicate for bug 52173. It it can also be, that "seeing many OUs" creates an additional slowdown.
Thanks for the info, we'll try the workaround of the related bug and tell you if it helps. Regarding relevance: this refers to the usecase, where school objects are imported by a central instance (e.g. "Schulträger"), which I think is valid for other environments as well.
(In reply to Oliver Friedrich from comment #3) > Regarding relevance: this refers to the usecase, where school objects are > imported by a central instance (e.g. "Schulträger"), which I think is valid > for other environments as well. So at that Schulträger, an admin logs in and makes X*Y imports through the UMC (with X number of schools and Y user roles)?
When I reduce the import job list with the workaround described in the related Bug like this: python /usr/share/pyshared/ucsschool/http_api/manage.py shell Python 2.7.13 (default, Apr 16 2021, 14:02:03) [GCC 6.3.0 20170516] on linux2 Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> from ucsschool.http_api.import_api.models import UserImportJob >>> UserImportJob.objects.last() <UserImportJob: UserImportJob 65 | schule004 (Finished)> >>> UserImportJob.objects.count() 65 >>> UserImportJob.objects.filter(id__lt=65).delete() (64, {u'import_api.UserImportJob': 64}) >>> UserImportJob.objects.count() 1 I still get a long waiting time loading the module. This leads to the conclusion that the import module performance is also affected by a high number of OUs
(In reply to Daniel Tröder from comment #4) > (In reply to Oliver Friedrich from comment #3) > > Regarding relevance: this refers to the usecase, where school objects are > > imported by a central instance (e.g. "Schulträger"), which I think is valid > > for other environments as well. > So at that Schulträger, an admin logs in and makes X*Y imports through the > UMC (with X number of schools and Y user roles)? Yes - there is a special group of "global import admins" for this purpose No - the role is chosen in the import dialogue, which is atm "student" or "teacher" In the /var/log/univention/ucs-school-import/gunicorn_error.log I see GET-Requests for all schools I already imported to an the limbo_ou, like this: [2021-06-03 18:24:07 +0000] [87713] [DEBUG] GET /v1/schools/ [2021-06-03 18:24:14 +0000] [87713] [DEBUG] GET /v1/schools/ [2021-06-03 18:24:22 +0000] [87713] [DEBUG] GET /v1/schools/ [2021-06-03 18:24:29 +0000] [87713] [DEBUG] GET /v1/imports/users/ [2021-06-03 18:24:29 +0000] [87713] [DEBUG] GET /v1/schools/schule004/ [2021-06-03 18:24:30 +0000] [87713] [DEBUG] GET /v1/schools/DeaktivierteKonten/ [2021-06-03 18:24:31 +0000] [87713] [DEBUG] GET /v1/schools/DeaktivierteKonten/roles/
Thanks for the testing (comments 5 and 6). This clearly indicates a linear slowdown with the number of OUs. (In reply to Oliver Friedrich from comment #6) > (In reply to Daniel Tröder from comment #4) > > (In reply to Oliver Friedrich from comment #3) > > > Regarding relevance: this refers to the usecase, where school objects are > > > imported by a central instance (e.g. "Schulträger"), which I think is valid > > > for other environments as well. > > So at that Schulträger, an admin logs in and makes X*Y imports through the > > UMC (with X number of schools and Y user roles)? > > Yes - there is a special group of "global import admins" for this purpose > No - the role is chosen in the import dialogue, which is atm "student" or > "teacher" So yes: this makes <num schools> * 2 imports for such an admin. (With len(["student", "teacher"]) == 2) If the customer has all data of those schools and roles available, why don't they do the cli-import for all schools and roles at once? It'll be much faster and less error prone.
> If the customer has all data of those schools and roles available, why don't > they do the cli-import for all schools and roles at once? > It'll be much faster and less error prone. Sadly, the customer has some more accessibility levels, which clearly separate shell access from import permissions. So the standard import user has only UMC access for importing all schools.
The performance issue still occurs with UCS5.0-2 and ucsschool=5.0 v3 An "all-school"-admin needs 1min50s to open the Import-UMC-Module for 1179 schools an decent-powered (Amazon EC2 instance with more IOPS) system.