Univention Bugzilla – Bug 42105
create script to import test users
Last modified: 2016-10-04 13:24:52 CEST
The script should create a CSV file and optionally start the import. Options: - number of students, teachers, staff, teachers+staff to create (unique, readable names) - number of school classes to create (unique, readable names) - list of schools (OUs) to put users into - whether to automatically start the import of the CSV file -- (rest of line) options to pass to import script If len(OUs) > 1: 50% of teachers, staff, teachers+staff will be put into two schools Users will be (more or less) evenly distributed among classes and schools.
r71930: code r71932: advisory usage: ucs-school-testuser-import [-h] [-c CSVFILE] [-n] [--staff STAFF] [--students STUDENTS] [--teachers TEACHERS] [--staffteachers STAFFTEACHERS] [--classes CLASSES] [--inclasses INCLASSES] [--schools SCHOOLS] [-v] OU [OU ...] ... UCS@school import test users tool positional arguments: OU Name of school (OU) the users will be put into. ImportArguments All arguments following the OU names (first one must start with a dash) will be passed to the import script. optional arguments: -h, --help show this help message and exit -c CSVFILE, --csvfile CSVFILE File to write to [default: test_users_%Y-%m-%d_%H:%M:%S.csv]. -n, --nostart Disable automatic start of import script [default: False]. --staff STAFF Number of staff to create [default: 0]. --students STUDENTS Number of students to create [default: 0]. --teachers TEACHERS Number of teachers to create [default: 0]. --staffteachers STAFFTEACHERS Number of staff_and_teacher users to create [default: 0]. --classes CLASSES Number of classes to create [default: 0]. --inclasses INCLASSES Number of classes to put users (except students) into per school [default: 2]. --schools SCHOOLS Number of schools to put users (except students) into (if #OUs > 1) [default: 2]. -v, --verbose Enable debugging output on the console [default: False]. IMPORTANT: If you wish to add more users to a system that has already imported some, you must append "--no-delete" to the cmdline after the list of OUs, or the previous users will be deleted. BTW: That way you can also pass "--dry-run" to the import script.
The import failed with the following error message: Success adding ImportStudent(name='e.brinkman2', school='School2', dn='uid=e.brinkman2,cn=schueler,cn=users,ou=School2,dc=autotest300,dc=local') (source_uid:NewDB record_uid: Elfrun.Brinkman2). Adding ImportStudent(name='e.brinkmann2', school='School1', dn='uid=e.brinkmann2,cn=schueler,cn=users,ou=School1,dc=autotest300,dc=local', old_dn=None) (source_uid:NewDB record_uid:Elftraud.Brinkmann2) attributes={'$dn$': 'uid=e.brinkmann2,cn=schueler,cn=users,ou=School1,dc=autotest300,dc=local', 'display_name': 'Elftraud Brinkmann2', 'record_uid': u'Elftraud.Brinkmann2', 'firstname': 'Elftraud', 'lastname': 'Brinkmann2', 'type_name': 'Student', 'school': 'School1', 'name': 'e.brinkmann2', 'disabled': 'none', 'email': u'elftraud.brinkmann2@autotest300.local', 'birthday': None, 'type': 'importStudent', 'schools': ['School1'], 'password': '7_#theVQ8JkuL(l', 'source_uid': u'NewDB', 'school_classes': {'School1': ['School1-1', 'School1-0', 'School1-q']}, 'objectType': 'users/user'} udm_properties={u'phone': [u'+55-938-97591'], 'overridePWHistory': '1', u'description': u'A student.', 'overridePWLength': '1'}... Creating ImportStudent(name='e.brinkmann2', school='School1', dn='uid=e.brinkmann2,cn=schueler,cn=users,ou=School1,dc=autotest300,dc=local', old_dn=None) ImportStudent(name='e.brinkmann2', school='School1', dn='uid=e.brinkmann2,cn=schueler,cn=users,ou=School1,dc=autotest300,dc=local') successfully created Success adding ImportStudent(name='e.brinkmann2', school='School1', dn='uid=e.brinkmann2,cn=schueler,cn=users,ou=School1,dc=autotest300,dc=local') (source_uid:NewDB record_uid: Elftraud.Brinkmann2). Username 'e.brinkmann-hartmann2[counter2]' to long, shortened to 'e.brinkmann-'. Adding ImportStudent(name='e.brinkmann-', school='School2', dn='uid=e.brinkmann-,cn=schueler,cn=users,ou=School2,dc=autotest300,dc=local', old_dn=None) (source_uid:NewDB record_uid:Elftrud.Brinkmann-Hartmann2) attributes={'$dn$': 'uid=e.brinkmann-,cn=schueler,cn=users,ou=School2,dc=autotest300,dc=local', 'display_name': 'Elftrud Brinkmann-Hartmann2', 'record_uid': u'Elftrud.Brinkmann-Hartmann2', 'firstname': 'Elftrud', 'lastname': 'Brinkmann-Hartmann2', 'type_name': 'Student', 'school': 'School2', 'name': 'e.brinkmann-', 'disabled': 'none', 'email': u'elftrud.brinkmann-hartmann2@autotest300.local', 'birthday': None, 'type': 'importStudent', 'schools': ['School2'], 'password': 't@FdXlF%6a%*44.', 'source_uid': u'NewDB', 'school_classes': {'School2': ['School2-1', 'School2-a']}, 'objectType': 'users/user'} udm_properties={u'phone': [u'+69-505-666960'], 'overridePWHistory': '1', u'description': u'A student.', 'overridePWLength': '1'}... Creating ImportStudent(name='e.brinkmann-', school='School2', dn='uid=e.brinkmann-,cn=schueler,cn=users,ou=School2,dc=autotest300,dc=local', old_dn=None) Entry #0: ValidationError when adding ImportStudent(name='e.brinkmann-', school='School2', dn='uid=e.brinkmann-,cn=schueler,cn=users,ou=School2,dc=autotest300,dc=local', old_dn=None) (source_uid:NewDB record_uid: Elftrud.Brinkmann-Hartmann2): {'name': ["Username must only contain numbers, letters and dots, and may not be 'admin'!"]} Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 131, in create_and_modify_users success = user.create(lo=self.connection) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 149, in create return super(ImportUser, self).create(lo, validate) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 417, in create success = self.create_without_hooks(lo, validate) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 152, in create_without_hooks success = super(ImportUser, self).create_without_hooks(lo, validate) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 430, in create_without_hooks raise ValidationError(self.errors.copy()) UserValidationError: ValidationError when adding ImportStudent(name='e.brinkmann-', school='School2', dn='uid=e.brinkmann-,cn=schueler,cn=users,ou=School2,dc=autotest300,dc=local', old_dn=None) (source_uid:NewDB record_uid: Elftrud.Brinkmann-Hartmann2): {'name': ["Username must only contain numbers, letters and dots, and may not be 'admin'!"]} http://jenkins.knut.univention.de:8080/job/UCSschool%204.1/job/UCSschool%204.1%20(R2)%20Large%20Environment/ws/autotest-300-ucsschool-large.log
(In reply to Stefan Gohmann from comment #2) > {'name': ["Username must only contain numbers, > letters and dots, and may not be 'admin'!"]} Fixed in separate Bug #42313. → ucs-school-import 14.0.16-10.307.201609081119
@StefanG: does the script what you expected/need? If yes, @Florian: please check if the script is correctly installed and usable on your test machine. It should be possible to import additional test users later on via the script. So if your test instance is missing of 1000 additional users, it should be possible to import these users via ucs-school-testuser-import.
(In reply to Sönke Schwardt-Krummrich from comment #4) > @StefanG: does the script what you expected/need? Good question. I'm still unable to test it because the import always fails, for example: http://jenkins.knut.univention.de:8080/job/UCSschool%204.1/job/UCSschool%204.1%20(R2)%20Large%20Environment/lastSuccessfulBuild/artifact/autotest-300-ucsschool-large.log Success adding ImportStudent(name='d.potts', school='School1', dn='uid=d.potts,cn=schueler,cn=users,ou=School1,dc=autotest300,dc=local') (source_uid:NewDB record_uid: Dix.Potts). Adding ImportStudent(name='d.pove...', school='School2', dn='uid=d.pove...,cn=schueler,cn=users,ou=School2,dc=autotest300,dc=local', old_dn=None) (source_uid:NewDB record_uid:Djamal.Pove...) attributes={'$dn$': 'uid=d.pove...,cn=schueler,cn=users,ou=School2,dc=autotest300,dc=local', 'display_name': 'Djamal Pove...', 'record_uid': u'Djamal.Pove...', 'firstname': 'Djamal', 'lastname': 'Pove...', 'type_name': 'Student', 'school': 'School2', 'name': 'd.pove...', 'disabled': 'none', 'email': u'djamal.pove...@autotest300.local', 'birthday': None, 'type': 'importStudent', 'schools': ['School2'], 'password': 'gJS2@F5w/tY?5PQ', 'source_uid': u'NewDB', 'school_classes': {'School2': ['School2-1', 'School2-0', 'School2-l']}, 'objectType': 'users/user'} udm_properties={u'phone': [u'+65-020-791739'], 'overridePWHistory': '1', u'description': u'A student.', 'overridePWLength': '1'}... Creating ImportStudent(name='d.pove...', school='School2', dn='uid=d.pove...,cn=schueler,cn=users,ou=School2,dc=autotest300,dc=local', old_dn=None) Entry #0: ValidationError when adding ImportStudent(name='d.pove...', school='School2', dn='uid=d.pove...,cn=schueler,cn=users,ou=School2,dc=autotest300,dc=local', old_dn=None) (source_uid:NewDB record_uid: Djamal.Pove...): {'name': ["Username must only contain numbers, letters and dots, and may not be 'admin'!"]} Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 131, in create_and_modify_users success = user.create(lo=self.connection) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 149, in create return super(ImportUser, self).create(lo, validate) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 417, in create success = self.create_without_hooks(lo, validate) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 152, in create_without_hooks success = super(ImportUser, self).create_without_hooks(lo, validate) File "/usr/lib/pymodules/python2.7/ucsschool/lib/models/base.py", line 430, in create_without_hooks raise ValidationError(self.errors.copy()) UserValidationError: ValidationError when adding ImportStudent(name='d.pove...', school='School2', dn='uid=d.pove...,cn=schueler,cn=users,ou=School2,dc=autotest300,dc=local', old_dn=None) (source_uid:NewDB record_uid: Djamal.Pove...): {'name': ["Username must only contain numbers, letters and dots, and may not be 'admin'!"]} Starting univention-directory-notifier Starting Univention Directory Notifier daemon. ok: run: univention-directory-notifier: (pid 20457) 1s, normally down done. univention-directory-notifier started More than 0 errors. Exiting. Errors: 0: ValidationError when adding ImportStudent(name='d.pove...', school='School2', dn='uid=d.pove...,cn=schueler,cn=users,ou=School2,dc=autotest300,dc=local', old_dn=None) (source_uid:NewDB record_uid: Djamal.Pove...): {'name': ["Username must only contain numbers, letters and dots, and may not be 'admin'!"]} 2016-09-10 19:24:59 INFO ucs-school-testuser-import.main:244 ------ ucs-school-user-import return with exit code 1 ------ *** Failed 1: /usr/share/ucs-school-import/scripts/ucs-school-testuser-import --students 20000 --classes 2000 --teachers 800 --inclasses 3 -v School1 School2 School3
Please use "ucs-school-import" as sourceUID instead of "NewDB" (which has no meaning at all).
(In reply to Florian Best from comment #6) > Please use "ucs-school-import" as sourceUID instead of "NewDB" (which has no > meaning at all). r72707: The default ID for test user imports has been changed to "TESTID". To change the sourceUID for an import job, please append "--sourceUID=<myID>" as argument to the script after listing the OUs. If just adding more users (not deleting those from previous runs), "--no-delete" may be the better solution.
(In reply to Stefan Gohmann from comment #5) > 0: ValidationError when adding ImportStudent(name='d.pove...', > school='School2', > dn='uid=d.pove...,cn=schueler,cn=users,ou=School2,dc=autotest300,dc=local', > old_dn=None) (source_uid:NewDB record_uid: Djamal.Pove...): {'name': > ["Username must only contain numbers, letters and dots, and may not be > 'admin'!"]} Such a username should since ucs-school-import 14.0.16-15 (r72474) not be possible anymore. ATM my internet doesn't allow me to read large logfiles. Please check the version of ucs-school-import used in the test.
(In reply to Daniel Tröder from comment #8) > Such a username should since ucs-school-import 14.0.16-15 (r72474) not be > possible anymore. > > ATM my internet doesn't allow me to read large logfiles. Please check the > version of ucs-school-import used in the test. OK, I've checked the logs. The version is older. I've restarted the test.
On a completely fresh UCS@school system I get the following error and the import didn't work: root@xen7:~# ucs-school-testuser-import --classes 3 --teachers 5 --students 20 oldschool ------ Creating user information... ------ Created 3 class names for 1 schools. ------ Writing user information to CSV file 'test_users_2016-09-23_13:06:39.csv'... ------ Created 20 students. Created 5 teachers. Created a total of 25 users. ------ Starting import of CSV file: 'test_users_2016-09-23_13:06:39.csv'... ------ Running: /usr/share/ucs-school-import/scripts/ucs-school-user-import -c /usr/share/ucs-school-import/configs/ucs-school-testuser-import.json -i test_users_2016-09-23_13:06:39.csv ------ UCS@school import tool starting ------ Reading configuration from '/usr/share/ucs-school-import/configs/global_defaults.json'... Reading configuration from '/var/lib/ucs-school-import/configs/global.json'... Reading configuration from '/usr/share/ucs-school-import/configs/user_import_defaults.json'... Reading configuration from '/var/lib/ucs-school-import/configs/user_import.json'... Reading configuration from '/usr/share/ucs-school-import/configs/ucs-school-testuser-import.json'... ------ UCS@school import tool configured ------ Used configuration files: ['/usr/share/ucs-school-import/configs/global_defaults.json', '/var/lib/ucs-school-import/configs/global.json', '/usr/share/ucs-school-import/configs/user_import_defaults.json', '/var/lib/ucs-school-import/configs/user_import.json', '/usr/share/ucs-school-import/configs/ucs-school-testuser-import.json']. Using command line arguments: {'input': {'filename': 'test_users_2016-09-23_13:06:39.csv'}} Configuration is: {u'activate_new_users': {u'default': True}, u'classes': {u'reader': u'ucsschool.importer.reader.test_csv_reader.TestCsvReader'}, u'csv': {u'header_lines': 1, u'incell-delimiter': {u'default': u','}, u'mapping': {u'Benutzertyp': u'__type', u'Beschreibung': u'description', u'Klassen': u'school_classes', u'Nachname': u'lastname', u'Schulen': u'schools', u'Telefon': u'phone', u'Vorname': u'firstname'}}, u'dry_run': False, u'factory': u'ucsschool.importer.default_user_import_factory.DefaultUserImportFactory', u'input': {u'filename': 'test_users_2016-09-23_13:06:39.csv', u'type': u'csv'}, u'logfile': u'/var/log/univention/ucs-school-import.log', u'maildomain': None, u'mandatory_attributes': [u'firstname', u'lastname', u'name', u'school'], u'no_delete': False, u'output': {u'new_user_passwords': None, u'user_import_summary': u'/var/lib/ucs-school-import/user_import_summary_%Y-%m-%d_%H:%M:%S.csv'}, u'password_length': 15, u'scheme': {u'email': u'<firstname>.<lastname>@<maildomain>', u'recordUID': u'<firstname>.<lastname>', u'username': {u'allow_rename': False, u'default': u'<:umlauts><firstname>[0].<lastname><:lower>[COUNTER2]'}}, u'school': None, u'sourceUID': u'TESTID', u'tolerate_errors': 0, u'user_deletion': {u'delete': True, u'expiration': 0}, u'user_role': None, u'verbose': False} DefaultUserImportFactory.make_reader is now <class 'ucsschool.importer.reader.test_csv_reader.TestCsvReader'>. ------ Starting mass import... ------ Stopping univention-directory-notifier Stopping univention-directory-notifier daemon: . ok: down: univention-directory-notifier: 0s done. univention-directory-notifier stopped ------ Importing users... ------ ------ Starting to read users from input data... ------ Done reading 1. user: ImportStudent(name=None, school=None, dn=None) Done reading 2. user: ImportStudent(name=None, school=None, dn=None) Done reading 3. user: ImportStudent(name=None, school=None, dn=None) Done reading 4. user: ImportStudent(name=None, school=None, dn=None) Done reading 5. user: ImportStudent(name=None, school=None, dn=None) Done reading 6. user: ImportStudent(name=None, school=None, dn=None) Done reading 7. user: ImportStudent(name=None, school=None, dn=None) Done reading 8. user: ImportStudent(name=None, school=None, dn=None) Done reading 9. user: ImportStudent(name=None, school=None, dn=None) Done reading 10. user: ImportStudent(name=None, school=None, dn=None) Done reading 11. user: ImportStudent(name=None, school=None, dn=None) Done reading 12. user: ImportStudent(name=None, school=None, dn=None) Done reading 13. user: ImportStudent(name=None, school=None, dn=None) Done reading 14. user: ImportStudent(name=None, school=None, dn=None) Done reading 15. user: ImportStudent(name=None, school=None, dn=None) Done reading 16. user: ImportStudent(name=None, school=None, dn=None) Done reading 17. user: ImportStudent(name=None, school=None, dn=None) Done reading 18. user: ImportStudent(name=None, school=None, dn=None) Done reading 19. user: ImportStudent(name=None, school=None, dn=None) Done reading 20. user: ImportStudent(name=None, school=None, dn=None) Done reading 21. user: ImportTeacher(name=None, school=None, dn=None) Done reading 22. user: ImportTeacher(name=None, school=None, dn=None) Done reading 23. user: ImportTeacher(name=None, school=None, dn=None) Done reading 24. user: ImportTeacher(name=None, school=None, dn=None) Done reading 25. user: ImportTeacher(name=None, school=None, dn=None) ------ Read 25 users from input data. ------ ------ Detecting which users to delete... ------ ------ Deleting 0 users... ------ ------ Deleted 0 users. ------ ------ Creating / modifying users... ------ Entry #2: Could not retrieve mail domain from configuration nor from UCRV mail/hosteddomains. Traceback (most recent call last): File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 106, in create_and_modify_users user = self.determine_add_modify_action(imported_user) File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 195, in determine_add_modify_action imported_user.prepare_all(new_user=True) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 227, in prepare_all self.prepare_attributes(new_user) File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 249, in prepare_attributes self.make_email() File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 381, in make_email "mail/hosteddomains.", entry=self.entry_count, import_user=self) MissingMailDomain: Could not retrieve mail domain from configuration nor from UCRV mail/hosteddomains. Starting univention-directory-notifier Starting Univention Directory Notifier daemon. ok: run: univention-directory-notifier: (pid 28677) 0s, normally down done. univention-directory-notifier started More than 0 errors. Exiting. Errors: 2: Could not retrieve mail domain from configuration nor from UCRV mail/hosteddomains. ------ ucs-school-user-import return with exit code 1 ------ root@xen7:~# echo $? 1 root@xen7:~# ucr get mail/hosteddomains root@xen7:~# ucr get server/role domaincontroller_master
Can you set the username scheme for students so that the firstname is used instead of the lastname? This is like the old tool which created users like anton1 and teachers like g.krause1.
(In reply to Florian Best from comment #10) > On a completely fresh UCS@school system I get the following error and the > import didn't work: [..] > 2: Could not retrieve mail domain from configuration nor from UCRV > mail/hosteddomains. It's not the import scripts job to create mail domains. The user must either set the maildomain in the configuration (--set maildomain=...) or per UCR. r72785: students usernames are their firstnames The user can also set the desired schema on the cmdline with "--set scheme:username:student=<:umlauts><firstname><:lower>[COUNTER2]".
(In reply to Daniel Tröder from comment #12) > (In reply to Florian Best from comment #10) > > On a completely fresh UCS@school system I get the following error and the > > import didn't work: > [..] > > 2: Could not retrieve mail domain from configuration nor from UCRV > > mail/hosteddomains. > > It's not the import scripts job to create mail domains. The user must either > set the maildomain in the configuration (--set maildomain=...) or per UCR. Mail addresses are optional in UCS@school. The script current makes them mandatory. Please add an option that specifies, if an email should be automatically set for each imported user.
I will create a cmdline switch and a workaround for Bug #42503.
r72807: made email address creation optional (default: off) # /usr/share/ucs-school-import/scripts/ucs-school-testuser-import -h usage: ucs-school-testuser-import [-h] [-c CSVFILE] [-n] [--staff STAFF] [--students STUDENTS] [--teachers TEACHERS] [--staffteachers STAFFTEACHERS] [--classes CLASSES] [--inclasses INCLASSES] [--schools SCHOOLS] [--email] [-v] OU [OU ...] ... [..] --email Create email addresses for users [default: False]. [..] When using --email either an email domain must be created in the corresponding UMC-module or must be configured on the command line with --set maildomain=...
Please also build the package before when setting the bug to fixed. I will build it now.
Please rename --email into --create-mail-addresses.
(In reply to Florian Best from comment #17) > Please rename --email into --create-mail-addresses. done: r72859
(In reply to Daniel Tröder from comment #19) > (In reply to Florian Best from comment #17) > > Please rename --email into --create-mail-addresses. > done: r72859 I will rebuild the package.
The help/prolog contains still the old name: When using --email either an email domain must be created in the corresponding UMC-module or must be configured on the command line with --set maildomain=...
OK: creation without mail address OK: creation with mail address
r72896: fixed prolog
OK: prolog
UCS@school 4.1 R2 v5 has been released. http://docs.software-univention.de/changelog-ucsschool-4.1R2v5-de.html If this error occurs again, please clone this bug.