Univention Bugzilla – Bug 43936
UCS@school 4.2: ucs-school-import: lower memory usage of import
Last modified: 2017-04-11 17:45:49 CEST
This change is already part of UCS@school 4.2, but QA is not done yet. +++ This bug was initially created as a clone of Bug #41882 +++ A customers system with 32GB RAM could not run an import job with 70,000 users, without adding GBs of swap. Ideas: * invalidate ucsschool.lib cache at certain points (e.g. after delete phase) * remove ImportUser objects from memory after create() / modify() / delete() and keep only textual representations for the three exports of statistics, passwords and summary (or write those as we go). * delete Exception objects and keep only textual representations (most of the Exceptions carry relevant ImportUser objects)
r78650: add test for performance of new import script Created jenkins job: http://jenkins.knut.univention.de:8080/job/UCSschool%204.2%20Performance/job/UCSschool%204.2%20Performance%20NewImport%2010.000/
http://jenkins.knut.univention.de:8080/job/UCSschool%204.2%20Performance/job/UCSschool%204.2%20Performance%20NewImport%2010.000/lastBuild/ *** BEGIN *** ['/usr/bin/python', '21_new-import_10000'] *** *** 02_import-performance/21_new-import_10000 *** Import 10.000 users in 20 schools with the new import script and test the performance *** *** START TIME: 2017-04-04 08:32:18 *** Starting create OUs INFO: create OUs took 259 seconds (allowed time: 1800 seconds) Starting new user import *** import_users_new('-v --teachers 800 --staff 500 --staffteachers 200 --students 8500 --classes 400 --csvfile /tmp/import10000.csv School00cfhhd School01kkqnc School02a02rm School03ljhqa School04jcmbi School05ui3sk School06gpmel School07x2upt School08ydhez School09hxqba School10tyq6j School11daqpx School12ehu0u School13mwlfl School14fvbix School15veayv School16k7i4q School17zogyh School18bx13s School19pc80v') INFO: new user import took 1671 seconds (allowed time: 14400 seconds) Starting new user import sync to s4 INFO: new user import sync to s4 took 6859 seconds (allowed time: 14460 seconds) Starting UMC authentication INFO: UMC authentication took 4 seconds (allowed time: 8 seconds) Starting create test user INFO: create test user took 86 seconds (allowed time: 195 seconds) Starting samba4 auth INFO: samba4 auth took 1 seconds (allowed time: 20 seconds) INFO: user password reset took 89 seconds (allowed time: 395 seconds) INFO: Found 10007 OpenLDAP users INFO: Found 10010 Samba4 users *** END TIME: 2017-04-04 11:03:00 *** *** TEST DURATION (H:MM:SS.ms): 2:30:41.731477 ***
OK: code change OK: manual test OK: jenkins test
UCS@school 4.2 v1 has been released. http://docs.software-univention.de/release-notes-ucsschool-4.2v1-de.html If this error occurs again, please clone this bug.