Bug 41239 - create new ucs@school import-tool using the python-lib
create new ucs@school import-tool using the python-lib
Status: RESOLVED FIXED
Product: UCS@school
Classification: Unclassified
Component: Import scripts
unspecified
Other Linux
: P5 normal (vote)
: UCS@school 4.1 R2
Assigned To: Daniel Tröder
Sönke Schwardt-Krummrich
: interim-1
Depends on: 41243 41344 41345 41346 41348 41349 41350 41354
Blocks: 26478 41242 41269 41469 41470 41471 41472
  Show dependency treegraph
 
Reported: 2016-05-10 14:19 CEST by Daniel Tröder
Modified: 2016-09-30 11:57 CEST (History)
3 users (show)

See Also:
What kind of report is it?: Feature Request
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Tröder univentionstaff 2016-05-10 14:19:25 CEST
Meta-Bug
Comment 1 Daniel Tröder univentionstaff 2016-05-25 15:51:29 CEST
Current state (r69527):
* needs most current ucs-school-lib, Attributes source_uid (source DB ID) and record_uid (maps to a single user in source DB) have been added to ucs-school-libs ucsschool.lib.models.User
* supports import of schwardts example CSV files in new and legacy formats
* stores "historically unique usernames" in LDAP
* usernames can be generated with the same rules as in manual-4.1#users:templates with the addition of [ALWAYSCOUNTER] and [COUNTER2]
* support for new input formats, logic etc is added by subclassing and configuring
* legacy csv support was added this way
* the configuration is a singleton, read-only dict, that is created from JSON files (and the command line)
* adding a network frontend should be simple
* imports only users
* missing features/bugs: see blocking bugs


Historically unique usernames:
------------------------------
Below container cn=unique-usernames,cn=ucsschool,cn=univention,$ldap_base one object per username-base is created with an attribute 'ucsschoolUsernameNextNumber' that is an integer: the next free number.
-> username_handler = factory.make_username_handler()
-> username_handler.format_username()

Extending:
----------
The Abstract Factory pattern is used to decouple object creation from object use and at the same time enforce the use of objects that belong together.
To support new user import formats/logic the classes that have to be changed can be subclassed and then objects of those will be returned by a custom factory.
The factory to use is initialized through its fully dotted Python path. So it can be configured in the configuration file.

-> ucsschool.importer.factory.setup_factory()
-> factory = ucsschool.importer.factory.Factory()
-> reader = factory.make_reader()
-> import_user = factory.make_import_user()

Legacy CSV:
-----------
As an example of how to add support for a "new" input format, the support for the legacy CSV format was added to the package.
* A configuration file:
usr/share/ucs-school-import/configs/legacy.json
* Subclasses specializing some code:
modules/ucsschool/importer/legacy/legacy_import_user.py
modules/ucsschool/importer/legacy/legacy_csv_factory.py
modules/ucsschool/importer/legacy/legacy_user_import.py
modules/ucsschool/importer/legacy/legacy_csv_reader.py
* Run it with:
# /usr/share/ucs-school-import/scripts/ucs-school-import2 /usr/share/ucs-school-import/configs/legacy.json

Configuration:
--------------
The configuration is a singleton, read-only dict object. It must be initialized once with the path to a JSON file. It will first read a default configuration from /usr/share/ucs-school-import/configs/defaults.json and overwrite it with settings from the specified JSON file. Additional arguments can be supplied (e.g. command line options) to overwrite even those.
The configuration file is documented in /usr/share/doc/ucs-school-import/configuration_readme.txt
-> ucsschool.importer.configuration.setup_configuration()
-> config = ucsschool.importer.configuration.Configuration()

Network frontend:
-----------------
If in the future a network frontend should be added, a design decision has to be made:
Either use one Python interpreter (->fork) per import job (probably no code change needed) or change the configuration and factory singletons (and their use) to support one of them per job.

I'd suggest the first one, as it would make the API and the implementation of the master process simple. It would just need a table [job|conf|pid] to manage multiple simultanious import jobs.

Logging:
--------
u.i.utils.logging2udebug.get_logger() is used throughout the code to always get the same logger "ucsschool.import". It is a descendant of the ucs-schoo-libs logger "ucsschool". add_stdout_handler() additionally writes all messages (DEBUG) to stdout. add_file_handler() creates two configuration files, with the default configuration /var/log/univention/ucsschool-import.log (level DEBUG) and /var/log/univention/ucsschool-import.info (level INFO).

Workflow:
---------
1. initilize global configuration object -> u.i.configuration.setup_configuration()
2. configure logging -> u.i.utils.logging2udebug.{get_logger, add_stdout_handler, add_file_handler}()
3. initilize global factory object -> u.i.factory.setup_factory()
4. start mass import -> u.i.mass_import.mass_import.MassImport.mass_import()

4.1 user import -> u.i.mass_import.mass_import.MassImport.import_users()
* A subclass of u.i.reader.base_reader.BaseReader must be implemented to convert the input data into a list of u.i.models.import_user.ImportUser. For the new CSV format u.i.reader.csv_reader.CsvReader was implemented and for the legacy CSV format u.i.legacy.legacy_csv_reader.LegacyCsvReader.
* The classes u.i.models.import_user.Import{Staff, Student, Teacher, TeachersAndStaff} are used to store the raw input data and in a second step to run some logic and consistency tests on it. Subclass them to change the logic with which input data is processed, normalization or to add tests.
* Now missing users are added or existing ones modified incl. all possible UDM attributes. Subclass and implement create_and_modify_hook() to run code before or after creation/modification.
* Then it is determined which users should be deleted,
* and then they are deleted. Subclass and implement delete_hook() to run code before or after deletion.
* At the end a statistic and a result-CSV is generated.
Comment 2 Sönke Schwardt-Krummrich univentionstaff 2016-06-01 15:17:43 CEST
From 35ucs-school-import.inst:

univention-config-registry set \
	ucsschool/import/debug/werror?no || die

The UCR variable "ucsschool/import/debug/werror" does not match with "ucsschool/debug/werror" used in 
modules/ucsschool/importer/utils/logging2udebug.py.

I would suggest to also move this UCR variable assignment to 
ucs-school-import.postinst
Comment 3 Sönke Schwardt-Krummrich univentionstaff 2016-06-01 17:54:30 CEST
Please merge the binary packages "ucs-school-import" and "ucs-school-import2" into "ucs-school-import". I think there is no need for a separate package.

Why is there a dependency to "python2.7-dev"?

Regarding filesnames I would suggest the following:
1) the "import_user" symlink is removed
2) the new import tool (aka "ucs-school-import2") is put in place if 
   "import_user"

We can now discuss, if we want to rename "ucs-school-import" into something like "ucs-school-import-legacy". For future development this would be correct, but currently the remaining scripts "import_groups" and so on are not legacy since there is no callable replacement. RFC.
Comment 4 Daniel Tröder univentionstaff 2016-06-02 08:49:36 CEST
done: r69739

For a) backwards compatibility, b) our product maintenance is more obvious for customers and c) transparency, I would not rename the legacy ucs-school-import script, but gradually replace symlinks or alter customer code to call the ucs-school-import2 script directly.
Comment 5 Daniel Tröder univentionstaff 2016-06-06 14:13:07 CEST
With r69848, there are now 3 types of hooks, that are run before and after adding, modifying and deleting user objects:
* (A) arbitrary executables (with certain names) can be stored in directories below /usr/share/ucs-school-import/hooks/ and are run by ucs-school-lib
* (B) ucsschool.importer.mass_import.user_import.UserImport can be subclassed and pre_create_hook(), post_modify_hook() etc can be overridden
* (C) Python files containing subclasses of ucsschool.importer.utils.pyhook.PyHook can be stored in directories below /usr/share/ucs-school-import/pyhooks/ and are run by ucs-school-import (in UserImport)

The order in which hooks are executed is:

B → C → A → add/mod/del → A → C → B
Comment 6 Daniel Tröder univentionstaff 2016-06-08 17:11:03 CEST
All dependent and blocked bugs solved, closing this meta bug.
Comment 7 Daniel Tröder univentionstaff 2016-06-09 10:23:49 CEST
r70008:
* User-import specific classes and files are renamed to prevent name clashes with future additions of imports of other object types.
* Configuration files are read in a predefined order. Settings in later read
configuration files overwrite settings from prior ones. The order is:

1. /usr/share/ucs-school-import/configs/global_defaults.json (do not edit)
2. /var/lib/ucs-school-import/configs/global.json (edit this)
3. Module specific configuration from /usr/share/ucs-school-import/configs/ (do not edit)
4. Module specific configuration from /var/lib/ucs-school-import/configs/ (edit this)
5. Configuration file set on the command line.
6. Options set on the command line by --set and its aliases.

For example:

'/usr/share/ucs-school-import/configs/global_defaults.json'...
'/var/lib/ucs-school-import/configs/global.json'...
'/usr/share/ucs-school-import/configs/user_import_defaults.json'...
'/var/lib/ucs-school-import/configs/user_import.json'...
'/usr/share/ucs-school-import/configs/user_import_legacy_defaults.json'...
'/var/lib/ucs-school-import/configs/user_import_legacy.json'...
- optionally followed by a configuration file mentioned on the cmdline
- optionally followed by --set settings
Comment 8 Daniel Tröder univentionstaff 2016-06-09 10:40:50 CEST
The role(s) of imported users must be detect early, so that objects of the correct User-classes can be instantiated.

This is done in reader.get_roles() during reading of the input data. In legacy_csv_reader.get_roles() is an example how the roles can be extracted from input-data.

If the input-data does not provide the role(s), it can be configured on the command line. All users found in the input data will then be of the same type.

-u USER_ROLE, --user_role USER_ROLE
  Set this, if the source data contains users with only one role <student|staff|teacher|teacher_and_staff>
(shortcut for --set user_role=...) [default: None].
Comment 9 Sönke Schwardt-Krummrich univentionstaff 2016-06-12 22:31:05 CEST
The script is now called ucs-school-user-import.
Comment 10 Sönke Schwardt-Krummrich univentionstaff 2016-06-12 22:33:08 CEST
2016-06-01 12:22:26 INFO  cmdline.main:108  ------ UCS@school import tool configured ------
2016-06-01 12:22:26 INFO  cmdline.main:109  Used configuration files: ['/usr/share/ucs-school-import/configs/global_defaults.json', '/var/lib/ucs-school-import/configs/global.json', '/usr/share/ucs-school-import/configs/user_import_defaul
ts.json', '/var/lib/ucs-school-import/configs/user_import.json', 'import.json'].
2016-06-01 12:22:26 INFO  cmdline.main:110  Using command line arguments: {'input': {'filename': 'import.csv'}, 'verbose': True, 'infile': 'import.csv'}
2016-06-01 12:22:26 INFO  cmdline.main:111  Configuration is:
{u'activate_new_users': {u'default': True},              
 u'classes': {},                                         
 u'csv': {u'header_lines': 1,                            
          u'incell-delimiter': {u'default': u','},       
          u'mapping': {u'Beschreibung': u'description',  
                       u'E-Mail': u'email',              
                       u'Gruppen': u'school_classes',    
                       u'Nach': u'lastname',             
                       u'OUs': u'schools',               
                       u'Vor': u'firstname'}},           
 u'dry_run': False,                                      
 u'factory': u'ucsschool.importer.default_user_import_factory.DefaultUserImportFactory',
 'infile': 'import.csv',                                 
 u'input': {u'filename': 'import.csv', u'type': u'csv'}, 
 u'logfile': u'/var/log/univention/ucsschool-import.log',
 u'maildomain': None,                                    
 u'mandatory_attributes': [u'firstname', u'lastname', u'name', u'school'],
 u'no_delete': False,                                    
 u'outdated_users': {u'deactivate': False,               
                     u'delete': True,                    
                     u'waiting_period': 0},              
 u'output': {u'new_user_passwords': u'/var/lib/ucs-school-import/new_user_passwords.csv',
             u'user_import_summary': u'/var/lib/ucs-school-import/user_import_summary.csv'},
 u'password_length': 15,                                 
 u'scheme': {u'email': u'<firstname>[0].<lastname>[COUNTER2]@<maildomain>',
             u'rid': u'<email>',                         
             u'username': {u'allow_rename': False,       
                           u'default': u'<:umlauts><firstname>[0].<lastname>[COUNTER2]'}},
 u'school': None,                                        
 u'sourceUID': u'FirstDB',                               
 u'tolerate_errors': 0,                                  
 u'user_deletion': {u'delete': True, u'expiration': 0},  
 u'user_role': u'student',                               
 u'verbose': True}                                       
2016-06-01 12:22:26 INFO  cmdline.do_import:91  ------ Starting mass import... ------
2016-06-01 12:22:26 WARNING utils.stopped_notifier:168  Stopping univention-directory-notifier
2016-06-01 12:22:31 INFO  utils._run:162  Stopping univention-directory-notifier daemon: .
ok: down: univention-directory-notifier: 0s              
done.                                                    
                                                         
2016-06-01 12:22:31 INFO  utils.stopped_notifier:180  univention-directory-notifier stopped
2016-06-01 12:22:31 INFO  mass_import.import_users:93  ------ Importing users... ------
2016-06-01 12:22:31 INFO  pyhooks_loader.get_plugins:53  Searching for plugins in: /usr/share/ucs-school-import/pyhooks...
2016-06-01 12:22:31 INFO  pyhooks_loader.get_plugins:80  Found plugins: {}
2016-06-01 12:22:31 INFO  user_import.read_input:79  ------ Starting to read users from input data... ------
2016-06-01 12:22:31 DEBUG base_reader.next:73  Input 2: ['Meyer', '2345', 'Adalbert', 'gsmitte', 'gsmitte-1A', 'Mein erster Schueler', 'addi@nstx.local'] -> {u'Testmuell': u'2345', u'Gruppen': u'gsmitte-1A', u'Nach': u'Meyer', u'Beschreibung': u'Mein erster Schueler', u'E-Mail': u'addi@nstx.local', u'Vor': u'Adalbert', u'OUs': u'gsmitte'}
2016-06-01 12:22:31 DEBUG base.cache:209  Initializing ('SchoolClass', ('name', 'gsmitte-1A'), ('school', 'gsmitte'))
2016-06-01 12:22:31 DEBUG csv_reader.map:186  ImportStudent(name=None, school=None, dn=None) attributes={'$dn$': None, 'display_name': u'Adalbert Meyer', 'record_uid': None, 'firstname': u'Adalbert', 'lastname': u'Meyer', 'type_name': 'Student', 'school': None, 'name': None, 'disabled': None, 'email': u'addi@nstx.local', 'birthday': None, 'type': 'importStudent', 'schools': u'gsmitte', 'password': 'xxxxxxxxxx', 'source_uid': None, 'school_classes': {'gsmitte': ['gsmitte-1A']}, 'objectType': 'users/user'} udm_properties={u'description': u'Mein erster Schueler'}
2016-06-01 12:22:31 INFO  user_import.read_input:83  Done reading 1. user: ImportStudent(name=None, school=None, dn=None)
2016-06-01 12:22:31 INFO  user_import.read_input:91  ------ Read 1 users from input data. ------
2016-06-01 12:22:31 INFO  user_import.create_and_modify_users:109  ------ Creating / modifying users... ------
2016-06-01 12:22:31 DEBUG user_import.create_and_modify_users:112  Creating / modifying user ImportStudent(name=None, school=None, dn=None)...
2016-06-01 12:22:31 DEBUG base.get_only_udm_obj:851  Getting ImportStudent UDM object by filter: (&(objectClass=ucsschoolType)(ucsschoolSourceUID=FirstDB)(ucsschoolRecordUID=addi@nstx.local))
2016-06-01 12:22:31 WARNING utils.stopped_notifier:190  Starting univention-directory-notifier
2016-06-01 12:22:31 INFO  utils._run:162  Starting Univention Directory Notifier daemon.
ok: run: univention-directory-notifier: (pid 8873) 0s, normally down
done.

2016-06-01 12:22:31 INFO  utils.stopped_notifier:197  univention-directory-notifier started
2016-06-01 12:22:31 ERROR cmdline.main:134  Outer Exception catcher: ldapError('Server is unwilling to perform: no global superior knowledge',)
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/frontend/cmdline.py", line 114, in main
    self.do_import()
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/frontend/cmdline.py", line 92, in do_import
    importer.mass_import()
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/mass_import.py", line 69, in mass_import
    self.import_users()
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/mass_import.py", line 96, in import_users
    user_import.create_and_modify_users(imported_users)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 113, in create_and_modify_users
    user = self.determine_add_modify_action(imported_user)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/mass_import/user_import.py", line 201, in determine_add_modify_action
    imported_user.prepare_all(new_user=True)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 171, in prepare_all
    self.prepare_attributes(new_user)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 184, in prepare_attributes
    self.make_username()
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/models/import_user.py", line 422, in make_username
    self.name = self.username_handler.format_username(self.name)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/utils/username_handler.py", line 92, in format_username
    counter = func(res)
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/utils/username_handler.py", line 128, in counter2
    return self._counters(name_base, "")
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/utils/username_handler.py", line 138, in _counters
    self.add_to_ldap(name_base, "2")
  File "/usr/lib/pymodules/python2.7/ucsschool/importer/utils/username_handler.py", line 20, in add_to_ldap
    [("objectClass", "ucsschoolUsername"), ("ucsschoolUsernameNextNumber", str(first_number))]
  File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 406, in add
    raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg)
ldapError: Server is unwilling to perform: no global superior knowledge
Comment 11 Daniel Tröder univentionstaff 2016-06-13 10:51:32 CEST
(In reply to Sönke Schwardt-Krummrich from comment #10)
> ldapError: Server is unwilling to perform: no global superior knowledge
Fixed copy&paste error in r70106.
Comment 12 Sönke Schwardt-Krummrich univentionstaff 2016-06-13 11:54:07 CEST
--dry-run has been ignored:

root@master65:/usr/share/ucs-school-import/scripts/test# ../ucs-school-user-import -c import.json -i import.csv --dry-run -v
2016-06-01 00:45:48 INFO  cmdline.main:102  ------ UCS@school import tool starting ------
2016-06-01 00:45:48 INFO  configuration.read:59  Reading configuration from 
[...]
2016-06-01 00:45:48 INFO  cmdline.main:111  Configuration is:
{u'activate_new_users': {u'default': True},
 u'classes': {},
 u'csv': {u'header_lines': 1,
          u'incell-delimiter': {u'default': u','},
          u'mapping': {u'Beschreibung': u'description',
                       u'E-Mail': u'email',
                       u'Gruppen': u'school_classes',
                       u'Nach': u'lastname',
                       u'OUs': u'schools',
                       u'Vor': u'firstname'}},
 u'dry_run': False,
 u'factory': u'ucsschool.importer.default_user_import_factory.DefaultUserImportFactory',
[...]
Comment 13 Sönke Schwardt-Krummrich univentionstaff 2016-06-13 12:40:09 CEST
I created a new user but [COUNTER2] directly started from "2" instead of "" since the username did not exist before. And there was no LDAP object for username history:


 u'scheme': {u'email': u'<firstname>[0].<lastname>[COUNTER2]@<maildomain>',
             u'rid': '<username>|<firstname>|<lastname>|<email>',
             u'username': {u'allow_rename': False,
                           u'default': u'<:umlauts><firstname>[0].<lastname>[COUNTER2]'}},

[...]

2016-06-01 01:34:23 INFO  user_import.read_input:79  ------ Starting to read users from input data... ------
2016-06-01 01:34:23 DEBUG base_reader.next:73  Input 2: ['Musta', '2345', 'Maxi', 'gsmitte', 'gsmitte-1A', 'Mein zweiter Schueler', 'addi2@nstx.local'] -> {u'Testmuell': u'2345', u'Gruppen': u'gsmitte-1A', u'Nach': u'Musta', u'Beschreibun
g': u'Mein zweiter Schueler', u'E-Mail': u'addi2@nstx.local', u'Vor': u'Maxi', u'OUs': u'gsmitte'}
2016-06-01 01:34:23 DEBUG base.cache:209  Initializing ('SchoolClass', ('name', 'gsmitte-1A'), ('school', 'gsmitte'))
2016-06-01 01:34:23 DEBUG csv_reader.map:186  ImportStudent(name=None, school=None, dn=None) attributes={'$dn$': None, 'display_name': u'Maxi Musta', 'record_uid': None, 'firstname': u'Maxi', 'lastname': u'Musta', 'type_name': 'Student',
'school': None, 'name': None, 'disabled': None, 'email': u'addi2@nstx.local', 'birthday': None, 'type': 'importStudent', 'schools': u'gsmitte', 'password': 'xxxxxxxxxx', 'source_uid': None, 'school_classes': {'gsmitte': ['gsmitte-1A']}, 'objectType': 'users/user'} udm_properties={u'description': u'Mein zweiter Schueler'}
2016-06-01 01:34:23 INFO  user_import.read_input:83  Done reading 1. user: ImportStudent(name=None, school=None, dn=None)
2016-06-01 01:34:23 INFO  user_import.read_input:91  ------ Read 1 users from input data. ------
2016-06-01 01:34:23 INFO  user_import.create_and_modify_users:109  ------ Creating / modifying users... ------
2016-06-01 01:34:23 DEBUG user_import.create_and_modify_users:112  Creating / modifying user ImportStudent(name=None, school=None, dn=None)...
2016-06-01 01:34:23 DEBUG base.get_only_udm_obj:851  Getting ImportStudent UDM object by filter: (&(objectClass=ucsschoolType)(ucsschoolSourceUID=FirstDB)(ucsschoolRecordUID=|Maxi|Musta|addi2@nstx.local))
2016-06-01 01:34:23 INFO  user_import.create_and_modify_users:128  Adding ImportStudent(name='M.Musta2', school='gsmitte', dn='uid=M.Musta2,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', old_dn=None) (source_uid:FirstDB record_uid:|Maxi|Musta|addi2@nstx.local) attributes={'$dn$': 'uid=M.Musta2,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', 'display_name': 'Maxi Musta', 'record_uid': u'|Maxi|Musta|addi2@nstx.local', 'firstname': 'Maxi', 'lastname': 'Musta', 'type_name': 'Student', 'school': 'gsmitte', 'name': 'M.Musta2', 'disabled': 'none', 'email': u'addi2@nstx.local', 'birthday': None, 'type': 'importStudent', 'schools': ['gsmitte'], 'password': 'xxxxxxxxxx', 'source_uid': u'FirstDB', 'school_classes': {'gsmitte': ['gsmitte-1A']}, 'objectType': 'users/user'} udm_properties={'overridePWHistory': '1', u'description': u'Mein zweiter Schueler', 'overridePWLength': '1'}...
2016-06-01 01:34:23 DEBUG base.call_hooks:363  /usr/share/ucs-school-import/hooks/user_create_pre.d not found or empty.
2016-06-01 01:34:23 DEBUG base.get_only_udm_obj:851  Getting ImportStudent UDM object by filter: username=M.Musta2
2016-06-01 01:34:23 INFO  base.create_without_hooks:425  Creating ImportStudent(name='M.Musta2', school='gsmitte', dn='uid=M.Musta2,cn=schueler,cn=users,ou=gsmitte,dc=nstx,dc=local', old_dn=None)


slapcat diff contained:
+dn: cn=M.Musta,cn=unique-usernames,cn=ucsschool,cn=univention,dc=nstx,dc=local
+ucsschoolUsernameNextNumber: 3
+objectClass: ucsschoolUsername
+cn: M.Musta
Comment 14 Daniel Tröder univentionstaff 2016-06-13 14:51:00 CEST
(In reply to Sönke Schwardt-Krummrich from comment #12)
> --dry-run has been ignored:
The meaning of the --dry-run switch has changed in r70131 / 14.0.9-3. The default is now "False", meaning that without any cmdline/config setting, it will run a real import.
Comment 15 Daniel Tröder univentionstaff 2016-06-13 16:15:12 CEST
(In reply to Sönke Schwardt-Krummrich from comment #13)
> I created a new user but [COUNTER2] directly started from "2" instead of ""
> since the username did not exist before. And there was no LDAP object for
> username history:
> 
> 
>  u'scheme': {u'email': u'<firstname>[0].<lastname>[COUNTER2]@<maildomain>',
>              u'rid': '<username>|<firstname>|<lastname>|<email>',
>              u'username': {u'allow_rename': False,
>                            u'default':
> u'<:umlauts><firstname>[0].<lastname>[COUNTER2]'}},
> 

> slapcat diff contained:
> +dn:
> cn=M.Musta,cn=unique-usernames,cn=ucsschool,cn=univention,dc=nstx,dc=local
> +ucsschoolUsernameNextNumber: 3
> +objectClass: ucsschoolUsername
> +cn: M.Musta
This is probably because of a bug in user_import_defaults.json: "[COUNTER2]" should not be used in scheme→email, only in scheme→username. In previous versions the email address could be formatted in that way too. Maybe your version still supports this. With a current version, I cannot reproduce the problem. The variable was removed from the configuration file in r70143. Please update your copies in /var/lib/ucs-school-import/configs/.
Comment 16 Daniel Tröder univentionstaff 2016-06-13 16:25:02 CEST
r70144: The user-editable configuration files in /var/lib/... are now empty ("{}"), so changes in the defaults files (/var/share/ucs...) do actually matter.
Comment 17 Sönke Schwardt-Krummrich univentionstaff 2016-06-14 00:02:55 CEST
Starting import via ['/usr/share/ucs-school-import/scripts/ucs-school-user-import', '-c', '/tmp/34_import-users_via_cli_v2.P0zfX5/config.B6nD2j']


2016-06-01 12:56:59 INFO  cmdline.main:108  ------ UCS@school import tool configured ------
2016-06-01 12:56:59 INFO  cmdline.main:109  Used configuration files: ['/usr/share/ucs-school-import/configs/global_defaults.json', '/var/lib/ucs-school-import/configs/global.json', '/usr/share/ucs-school-import/configs/user_import_defaults.json', '/var/lib/ucs-school-import/configs/user_import.json', '/tmp/34_import-users_via_cli_v2.P0zfX5/config.B6nD2j'].
2016-06-01 12:56:59 INFO  cmdline.main:110  Using command line arguments: {'input': {'filename': '/var/lib/ucs-school-import/new-format-userimport.csv'}}

I think the last line is plain wrong and raises an error. The specified config file contained an input:filename setting the is overwritten by the "mystery" commandline default.
Comment 18 Sönke Schwardt-Krummrich univentionstaff 2016-06-14 01:41:01 CEST
Please rename "scheme:rid" into "scheme:recordUID". "rid" has nothing to do with a samba RID and irritates me every time reading it.
Comment 19 Daniel Tröder univentionstaff 2016-06-14 09:53:58 CEST
(In reply to Sönke Schwardt-Krummrich from comment #17)
> Starting import via
> ['/usr/share/ucs-school-import/scripts/ucs-school-user-import', '-c',
> '/tmp/34_import-users_via_cli_v2.P0zfX5/config.B6nD2j']

> 2016-06-01 12:56:59 INFO  cmdline.main:110  Using command line arguments:
> {'input': {'filename':
> '/var/lib/ucs-school-import/new-format-userimport.csv'}}
> 
> I think the last line is plain wrong and raises an error. The specified
> config file contained an input:filename setting the is overwritten by the
> "mystery" commandline default.
r70151: removed default "infile" cmdline argument

(In reply to Sönke Schwardt-Krummrich from comment #18)
> Please rename "scheme:rid" into "scheme:recordUID". "rid" has nothing to do
> with a samba RID and irritates me every time reading it.
r70151: renamed all occurences of "sid" to "sourceUID" and of "rid" to "recordUID".
Comment 21 Daniel Tröder univentionstaff 2016-06-15 11:34:45 CEST
(In reply to Florian Best from comment #20)
> The renaming seems to have failed:
> http://jenkins.knut.univention.de:8080/job/UCSschool%204.1/job/UCSschool%204.
> 1%20(R2)%20Multiserver/SambaVersion=s4-only-master/137/testReport/
> 90_ucsschool/70_users_module/test/
Fixed in r70193.