Bug 34199 - Add a UMC module for importing users from a CSV file
Add a UMC module for importing users from a CSV file
Status: CLOSED FIXED
Product: UCS@school
Classification: Unclassified
Component: UMC
UCS@school 3.2 R2
Other Linux
: P5 normal (vote)
: UCS@school 3.2 R2
Assigned To: Dirk Wiesenthal
Sönke Schwardt-Krummrich
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2014-02-26 00:54 CET by Dirk Wiesenthal
Modified: 2014-06-12 09:20 CEST (History)
2 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional): Release Goal
Max CVSS v3 score:


Attachments
page 2 of csv-import (52.78 KB, image/png)
2014-05-20 14:34 CEST, Sönke Schwardt-Krummrich
Details
page 3 of csv-import (50.87 KB, image/png)
2014-05-20 14:35 CEST, Sönke Schwardt-Krummrich
Details
page 4 of csv-import (55.35 KB, image/png)
2014-05-20 14:39 CEST, Sönke Schwardt-Krummrich
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dirk Wiesenthal univentionstaff 2014-02-26 00:54:30 CET
A new umc-module shall be added that enables admins to import many users at once
Comment 1 Sönke Schwardt-Krummrich univentionstaff 2014-04-30 11:52:47 CEST
As discussed there should be some changes:
1) The names of the 4 actions should be "Create"/"Erstellen", "Modify"/"Ändern", 
   "Delete"/"Löschen", "Ignore"/"Ignorieren"
2) The default button in the edit dialog should be renamed to "Apply/Übernehmen".
3) The edit dialog should not perform a syntax/required check of the widgets if 
   the action "ignore" has been selected. Otherwise it is not possible to ignore 
   broken lines without entering valid dummy values.
4) The user should be able to edit a cell by double clicking on it. In this case 
   the action cell should show a drop-down (not a string field!). When changing a 
   cell a syntax/requirement check has also to be performed (like in the edit 
   dialog).
   If this is not possible or too costly, I think a possible fallback would be, 
   that the edit dialog opens if the user double clicks on a cell. Ideally the 
   clicked cell is already focused in edit dialog.
5) To simplify ignoring lines, an addition button "Ignore" should be added above 
   the grid, that changes the action of selected lines to "Ignore".
Comment 2 Sönke Schwardt-Krummrich univentionstaff 2014-04-30 12:06:54 CEST
6) The module should detect different header lines with german and english 
   keywords.
7) The module should ignore empty CSV lines.
8) The module should check if the import would overrun the UCS license. According 
   to Stefan the AD takeover module does something similar.
Comment 3 Sönke Schwardt-Krummrich univentionstaff 2014-04-30 12:26:59 CEST
Optional wishes (only if "easy doing"):
9) The Escape key should close/abort the edit dialog
10) The grid often shows a horizontal slider without need. Is it possible to hide 
    the slider in such cases?
Comment 4 Dirk Wiesenthal univentionstaff 2014-05-01 00:51:01 CEST
(In reply to Sönke Schwardt-Krummrich from comment #1)
> As discussed there should be some changes:
> 1) The names of the 4 actions should be "Create"/"Erstellen",
> "Modify"/"Ändern", 
>    "Delete"/"Löschen", "Ignore"/"Ignorieren"

Done

> 2) The default button in the edit dialog should be renamed to
> "Apply/Übernehmen".

Done

> 3) The edit dialog should not perform a syntax/required check of the widgets
> if 
>    the action "ignore" has been selected. Otherwise it is not possible to
> ignore 
>    broken lines without entering valid dummy values.

Done

> 4) The user should be able to edit a cell by double clicking on it. In this
> case 
>    the action cell should show a drop-down (not a string field!). When
> changing a 
>    cell a syntax/requirement check has also to be performed (like in the
> edit 
>    dialog).
>    If this is not possible or too costly, I think a possible fallback would
> be, 
>    that the edit dialog opens if the user double clicks on a cell. Ideally
> the 
>    clicked cell is already focused in edit dialog.

Implemented the fallback: DblClick opens the dialog and focuses the cell attribute. cellEdit is a dojo grid feature that lacks a lot of functionality like date widgets, required=true, etc.

> 5) To simplify ignoring lines, an addition button "Ignore" should be added
> above 
>    the grid, that changes the action of selected lines to "Ignore".

Done

(In reply to Sönke Schwardt-Krummrich from comment #2)
> 6) The module should detect different header lines with german and english 
>    keywords.

Done. Note that this is done hard coded, e.g.:
  school_class = SchoolClassStringAttribute(_('Class'), aka=['Class', 'Klasse'])
Here a CSV column "Klasse" matches school_class even in en-US because of "aka". It is hard coded and does not work for other languages. This module with French localization would match "Classe" only in fr-FR - because of _('Class'). "aka" is of no help here.

> 7) The module should ignore empty CSV lines.

Done, also ignores something like ",,,,"

> 8) The module should check if the import would overrun the UCS license.
> According 
>    to Stefan the AD takeover module does something similar.

TBD

(In reply to Sönke Schwardt-Krummrich from comment #3)
> Optional wishes (only if "easy doing"):
> 9) The Escape key should close/abort the edit dialog

Sadly, not so easy. In theory this is just "closable: true" in the ConfirmDialog contructor. Unfortunately, I have done the dialog with dialog.confirmForm() which hides this constructor and returns a Deferred instead of the dialog so that the dialog is not manipulatable. Long story short: Would require reworking the whole dialog thing. I will have a look if time allows it at the end.

> 10) The grid often shows a horizontal slider without need. Is it possible to
> hide 
>     the slider in such cases?

Not so easy. It is already "overflow-x: auto;", so the scrollbar should appear only when necessary. It seems to be the y-scrollbar that appears, effectively reducing the overall width of the grid viewport and thus requiring the x-scrollbar to appear. I see no easy way to get this right. "overflow-y: auto;" would help only for small files.
Comment 5 Dirk Wiesenthal univentionstaff 2014-05-14 01:21:48 CEST
(In reply to Sönke Schwardt-Krummrich from comment #2)
> 8) The module should check if the import would overrun the UCS license.
> According 
>    to Stefan the AD takeover module does something similar.

Done. Last blocker. Setting RESOLVED for now
Comment 6 Sönke Schwardt-Krummrich univentionstaff 2014-05-20 14:33:14 CEST
Wording:
- "CSV import for UCS@school users" → "CSV import of UCS@school users"

- "CSV-Import für UCS@school-Benutzer"
   → "Benutzerverwaltung für Schulen über CSV-Dateien"

- "Delete those who are not mentioned in the uploaded file (only applies to users of the selected user role)"/"Nicht  in der hochgeladenen Datei genannten Benutzer löschen (bezieht sich nur  auf solche Benutzer mit der ausgewählten Benutzerrolle)"
  → "Austausch der vorhandenen $ROLLE durch die $ROLLE der CSV-Datei. Das bedeutet, dass alle $ROLLE, die nicht in der CSV-Datei vorhanden sind gelöscht werden." $ROLLE ist dynamisch durch die Benutzerrolle aus dem Dropdown der ersten Page zu ersetzen (also "Schüler", "Lehrer", "Lehrer/Mitarbeiter", "Mitarbeiter").

- Die Texte wurden angepasst. Siehe dazu die angehängten Screenshots!

- Keine Anführungszeichen im Beispiel (siehe Screenshot)

- "until a new license is installed" → "UCS license is imported"

- Es sollte deutlicher sein, dass im Grid "nur"! die ersten 10 Zeilen der 
  CSV-Datei angezeigt werden. → ist im Screenshot schon angepasst.

Bei falscher CSV-Datei wird kein Fehler angezeigt, stattdessen ein leeres Grid.
Klick auf zurück-Button funktioniert in dem Fall nicht (nichts passiert).
http://billy/~fbest/school3.png

Javascript exception in schoolcsvimport.js on line 542: recheck is undefined
this.standbyDuring(recheck);
happens when clicking on "ignore" every selected row

Bitte "Schule" aus der Spaltenauswahl entfernen. Ebenso "Deaktiviert".

Uncaught TypeError: Cannot read property 'focusNode' of undefined 
698:                        if (widget.focusNode) {
when doubleclicking on a column which doesn't exists as widget e.g. lineno.
Comment 7 Sönke Schwardt-Krummrich univentionstaff 2014-05-20 14:34:41 CEST
Created attachment 5916 [details]
page 2 of csv-import

Reordered widgets and changed texts
Comment 8 Sönke Schwardt-Krummrich univentionstaff 2014-05-20 14:35:33 CEST
Created attachment 5917 [details]
page 3 of csv-import

Changed text / please add some space between text and grid
Comment 9 Sönke Schwardt-Krummrich univentionstaff 2014-05-20 14:39:51 CEST
Created attachment 5918 [details]
page 4 of csv-import

changed text (see arrow :-)
Comment 10 Sönke Schwardt-Krummrich univentionstaff 2014-05-20 14:53:26 CEST
Problematic scenario: import the same CSV file twice: once as students and once as teachers. The CSV import module shows for all users a modification in the second import attempt.
The CSV import should show a conflict for each line where the username matches to an existing user object with a different role.
Comment 11 Florian Best univentionstaff 2014-05-20 15:37:41 CEST
When uploading a CVS-file which contains ISO8859-1(latin1) characters the following traceback occurs:

Die Ausführung des Kommandos schoolcsvimport/save ist fehlgeschlagen:

Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.6/univention/management/console/modules/__init__.py", line 204, in execute
    func( request )
  File "/usr/lib/pymodules/python2.6/univention/management/console/modules/decorators.py", line 591, in _response
    return function(self, request)
  File "/usr/lib/pymodules/python2.6/univention/management/console/modules/schoolcsvimport/__init__.py", line 168, in save_csv
    self.finished(request.id, [result])
  File "/usr/lib/pymodules/python2.6/univention/management/console/modules/__init__.py", line 276, in finished
    self.result( res )
  File "/usr/lib/pymodules/python2.6/univention/management/console/modules/__init__.py", line 283, in result
    self.signal_emit( 'success', response )
  File "/usr/lib/pymodules/python2.6/notifier/signals.py", line 75, in signal_emit
    self.__signals[ signal ].emit( *args )
  File "/usr/lib/pymodules/python2.6/notifier/signals.py", line 41, in emit
    cb( *args )
  File "/usr/lib/pymodules/python2.6/notifier/__init__.py", line 104, in __call__
    return self._function( *tmp, **self._kwargs )
  File "/usr/lib/pymodules/python2.6/univention/management/console/protocol/modserver.py", line 106, in _reply
    self.response( msg )
  File "/usr/lib/pymodules/python2.6/univention/management/console/protocol/modserver.py", line 305, in response
    data = str( msg )
  File "/usr/lib/pymodules/python2.6/univention/management/console/protocol/message.py", line 317, in __str__
    return Message._formattedMessage(self._id, self._type, self.mimetype, self.command, body, self.arguments)
  File "/usr/lib/pymodules/python2.6/univention/management/console/protocol/message.py", line 119, in _formattedMessage
    data = json.dumps( body )
  File "/usr/lib/pymodules/python2.6/simplejson/__init__.py", line 261, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/pymodules/python2.6/simplejson/encoder.py", line 214, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/pymodules/python2.6/simplejson/encoder.py", line 282, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 4: invalid continuation byte
Comment 12 Florian Best univentionstaff 2014-05-20 16:32:19 CEST
Currently this traceback occurs for every imported user (10.200.27.75).

20.05.14 05:00:45.310  MODULE      ( PROCESS ) : Going to create CSVStudent Wilhelmine11
20.05.14 05:00:45.317  MODULE      ( PROCESS ) : Creating CSVStudent(name='Wilhelmine11', school='Irgendeinname', dn='uid=Wilhelmine11,cn=schueler,cn=users,ou=Irgendeinname,dc=ucs,dc=school2')
20.05.14 05:00:45.320  MODULE      ( PROCESS ) : Creating MailDomain(name='ucs.school2', dn='cn=ucs.school2,cn=domain,cn=mail,dc=ucs,dc=school2')
20.05.14 05:00:45.320  MODULE      ( PROCESS ) : cn=ucs.school2,cn=domain,cn=mail,dc=ucs,dc=school2 already exists!
20.05.14 05:00:45.385  MODULE      ( PROCESS ) : Creating Group(name='Domain Users Irgendeinname', school='Irgendeinname', dn='cn=Domain Users Irgendeinname,cn=groups,ou=Irgendeinname,dc=ucs,dc=school2')
20.05.14 05:00:45.385  MODULE      ( PROCESS ) : cn=Domain Users Irgendeinname,cn=groups,ou=Irgendeinname,dc=ucs,dc=school2 already exists!
20.05.14 05:00:45.386  MODULE      ( PROCESS ) : Creating Group(name='Domain Users Irgendeinname', school='Irgendeinname', dn='cn=Domain Users Irgendeinname,cn=groups,ou=Irgendeinname,dc=ucs,dc=school2')
20.05.14 05:00:45.386  MODULE      ( PROCESS ) : cn=Domain Users Irgendeinname,cn=groups,ou=Irgendeinname,dc=ucs,dc=school2 already exists!
20.05.14 05:00:45.386  MODULE      ( PROCESS ) : Creating Group(name='Domain Users Irgendeinname', school='Irgendeinname', dn='cn=Domain Users Irgendeinname,cn=groups,ou=Irgendeinname,dc=ucs,dc=school2')
20.05.14 05:00:45.387  MODULE      ( PROCESS ) : cn=Domain Users Irgendeinname,cn=groups,ou=Irgendeinname,dc=ucs,dc=school2 already exists!
20.05.14 05:00:45.387  MODULE      ( PROCESS ) : Creating SchoolClass(name='Schule74-2A', school='Irgendeinname', dn='cn=Schule74-2A,cn=klassen,cn=schueler,cn=groups,ou=Irgendeinname,dc=ucs,dc=school2')
20.05.14 05:00:45.387  MODULE      ( PROCESS ) : cn=Schule74-2A,cn=klassen,cn=schueler,cn=groups,ou=Irgendeinname,dc=ucs,dc=school2 already exists!
20.05.14 05:00:45.388  MODULE      ( PROCESS ) : Creating ClassShare(name='Schule74-2A', school='Irgendeinname', dn='cn=Schule74-2A,cn=klassen,cn=shares,ou=Irgendeinname,dc=ucs,dc=school2')
20.05.14 05:00:45.393  MODULE      ( PROCESS ) : cn=Schule74-2A,cn=klassen,cn=shares,ou=Irgendeinname,dc=ucs,dc=school2 already exists!
20.05.14 05:00:45.394  MODULE      ( PROCESS ) : Creating Group(name='schueler-Irgendeinname', school='Irgendeinname', dn='cn=schueler-Irgendeinname,cn=groups,ou=Irgendeinname,dc=ucs,dc=school2')
20.05.14 05:00:45.394  MODULE      ( PROCESS ) : cn=schueler-Irgendeinname,cn=groups,ou=Irgendeinname,dc=ucs,dc=school2 already exists!
20.05.14 05:00:47.830  ADMIN       ( ERROR   ) : Post-modify operation failed:   File "/usr/lib/pymodules/python2.6/univention/admin/handlers/__init__.py", line 759, in _create
    self._ldap_post_create()

  File "/usr/lib/pymodules/python2.6/univention/admin/handlers/users/user.py", line 1851, in _ldap_post_create
    self.__update_groups()

  File "/usr/lib/pymodules/python2.6/univention/admin/handlers/users/user.py", line 1637, in __update_groups
    grpobj.fast_member_add( [ self.dn ], [ new_uid ] )

  File "/usr/lib/pymodules/python2.6/univention/admin/handlers/groups/group.py", line 451, in fast_member_add
    searchResult = self.lo.search(base=self.dn, attr=['uniqueMember','memberUid'])

  File "/usr/lib/pymodules/python2.6/univention/admin/uldap.py", line 347, in search
    raise univention.admin.uexceptions.noObject, _err2str(msg)

20.05.14 05:00:47.832  MODULE      ( WARN    ) : Something went wrong. Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.6/univention/management/console/modules/schoolcsvimport/util.py", line 207, in commit
    self.create(lo, validate=False)
  File "/usr/lib/pymodules/python2.6/ucsschool/lib/models/base.py", line 352, in create
    success = self.create_without_hooks(lo, validate)
  File "/usr/lib/pymodules/python2.6/ucsschool/lib/models/base.py", line 380, in create_without_hooks
    self.do_create(udm_obj, lo)
  File "/usr/lib/pymodules/python2.6/ucsschool/lib/models/user.py", line 176, in do_create
    success = super(User, self).do_create(udm_obj, lo)
  File "/usr/lib/pymodules/python2.6/ucsschool/lib/models/base.py", line 397, in do_create
    udm_obj.create()
  File "/usr/lib/pymodules/python2.6/univention/admin/handlers/__init__.py", line 333, in create
    return self._create()
  File "/usr/lib/pymodules/python2.6/univention/admin/handlers/__init__.py", line 765, in _create
    self.remove()
  File "/usr/lib/pymodules/python2.6/univention/admin/handlers/__init__.py", line 506, in remove
    return self._remove(remove_childs)
  File "/usr/lib/pymodules/python2.6/univention/admin/handlers/__init__.py", line 972, in _remove
    self._ldap_pre_remove()
  File "/usr/lib/pymodules/python2.6/univention/admin/handlers/users/user.py", line 2409, in _ldap_pre_remove
    self.sid=self.oldattr['sambaSID'][0]
KeyError: 'sambaSID'
Comment 13 Dirk Wiesenthal univentionstaff 2014-05-21 00:50:04 CEST
(In reply to Sönke Schwardt-Krummrich from comment #6)
> Wording:

Done

>   → "Austausch der vorhandenen $ROLLE durch die $ROLLE der CSV-Datei. Das
> bedeutet, dass alle $ROLLE, die nicht in der CSV-Datei vorhanden sind
> gelöscht werden." $ROLLE ist dynamisch durch die Benutzerrolle aus dem
> Dropdown der ersten Page zu ersetzen (also "Schüler", "Lehrer",
> "Lehrer/Mitarbeiter", "Mitarbeiter").

Setting the label dynamically is more difficult than it should be. I put the checkbox on the first page and used a more generic sentence. Also note that the label of the user role is not "Lehrer/Mitarbeiter" but "Lehrer und Mitarbeiter" which could lead to misunderstandings. Nothing infeasible, it is just easier the way I have done it now. If you feel it is not enough, let me know.

> - Es sollte deutlicher sein, dass im Grid "nur"! die ersten 10 Zeilen der 
>   CSV-Datei angezeigt werden. → ist im Screenshot schon angepasst.

Done

> 
> Bei falscher CSV-Datei wird kein Fehler angezeigt, stattdessen ein leeres
> Grid.
> Klick auf zurück-Button funktioniert in dem Fall nicht (nichts passiert).
> http://billy/~fbest/school3.png

Not reproducible. Empty CSV files now give an error message. But even before that change I was able to click "back". Example file?

> 
> Javascript exception in schoolcsvimport.js on line 542: recheck is undefined
> this.standbyDuring(recheck);
> happens when clicking on "ignore" every selected row

Done

> 
> Bitte "Schule" aus der Spaltenauswahl entfernen. Ebenso "Deaktiviert".

Done

> 
> Uncaught TypeError: Cannot read property 'focusNode' of undefined 
> 698:                        if (widget.focusNode) {
> when doubleclicking on a column which doesn't exists as widget e.g. lineno.

Done

(In reply to Sönke Schwardt-Krummrich from comment #10)
> Problematic scenario: import the same CSV file twice: once as students and
> once as teachers. The CSV import module shows for all users a modification
> in the second import attempt.
> The CSV import should show a conflict for each line where the username
> matches to an existing user object with a different role.

Done

(In reply to Florian Best from comment #11)
> When uploading a CVS-file which contains ISO8859-1(latin1) characters the
> following traceback occurs:

Done. UTF-8 and LATIN-1 supported. Anything incompatible will still raise. I would say this does not happen with real life files.


(In reply to Florian Best from comment #12)
> Currently this traceback occurs for every imported user (10.200.27.75).
> KeyError: 'sambaSID'

Seems to be a more generic error with UDM? Nothing done here until you reproduce it.
Comment 14 Sönke Schwardt-Krummrich univentionstaff 2014-05-21 12:34:23 CEST
Please move the checkbox back to page 2.

Checkbox text should be
"Austausch der vorhandenen Benutzer mit der ausgewählten Benutzerrolle durch die Benutzer der CSV-Datei. Das bedeutet, dass alle Benutzer, die nicht in der CSV-Datei enthalten sind, gelöscht werden."

The text changes of screenshot page 3 have not been merged.

If there is a user role conflict, the tooltip message is not (fully) translated:
"Benutzername:
It is not supported to change the role of a user. Schüler … cannot become a Lehrer".
Comment 15 Dirk Wiesenthal univentionstaff 2014-05-21 14:04:42 CEST
(In reply to Sönke Schwardt-Krummrich from comment #14)
> Please move the checkbox back to page 2.
> 
> Checkbox text should be
> "Austausch der vorhandenen Benutzer mit der ausgewählten Benutzerrolle durch
> die Benutzer der CSV-Datei. Das bedeutet, dass alle Benutzer, die nicht in
> der CSV-Datei enthalten sind, gelöscht werden."
> 
> The text changes of screenshot page 3 have not been merged.
> 
> If there is a user role conflict, the tooltip message is not (fully)
> translated:
> "Benutzername:
> It is not supported to change the role of a user. Schüler … cannot become a
> Lehrer".

Done
Comment 16 Sönke Schwardt-Krummrich univentionstaff 2014-05-22 17:34:47 CEST
(In reply to Dirk Wiesenthal from comment #13)

> (In reply to Florian Best from comment #12)
> > Currently this traceback occurs for every imported user (10.200.27.75).
> > KeyError: 'sambaSID'
> 
> Seems to be a more generic error with UDM? Nothing done here until you
> reproduce it.

This was probably a caching problem and should be fixed with bug 34924.
Comment 17 Sönke Schwardt-Krummrich univentionstaff 2014-05-22 17:37:11 CEST
CSV import module:
- valid headline keywords: 
  Username, First name, Last name, Birthday, Email, Password
- valid headline keywords: 
  Benutzername, Vorname, Nachname, Geburtstag, E-Mail, Passwort

*** Test1: (normal users)

Benutzername,Vorname,Nachname,Klasse
Dennis1,Dennis,Demo,12a
Daniela2,Daniela,Tester,13c
emil.langusername,Emil,Langusername,11b

→ import successful

*** Test2: (users with umlauts)

Benutzername,Vorname,Nachname,Klasse
Dennis1,Dennis,Demo,12a
Daniela2,Daniela,Tester,13c
emil.langusername,Emil,Langusername,11b
lydia,Lydia,Lüttich,12a
jörg.lämmers,Jörg Walter,Lämmers von der Weile,13c

→ first 3 lines are recognized as modification, the other 2 as new users
→ changed class of Daniela2 to 13d via edit dialog
→ 2 new users, Daniela2 switched to 13d, group "$OU-13d" has been created

*** Test3: (valid and invalid mail addresses)

Benutzername,Vorname,Nachname,Klasse,Email
Dennis1,Dennis,Demo,12a,dennis1@nstx.local
Daniela2,Daniela,Tester,13c,daniela2@nstx.local
emil.langusername,Emil,Langusername,11b,wrong@address.com
lydia,Lydia,Lüttich,12a
jörg.lämmers,Jörg Walter,Lämmers von der Weile,13c

→ 3 mail addresses have been added and one mail domain object

*** Test 4 (setting password and double class column)

Benutzername,Vorname,Nachname,Klasse,Passwort,Klasse
Dennis1,Dennis,Demo,12a,univention,12a
Daniela2,Daniela,Tester,13c,univention,12c
emil.langusername,Emil,Langusername,11b

→ Bug 34921

*** Test 5 (only first and last name)

Vorname,Nachname
Dennis,Demo
Daniela,Tester
Emil,Langusername
Zack,Zacharias
Deborah,Demonstration

→ Module found users Dennis1, Daniela2 and emil.langusername automatically (marked as modification)
→ 2 new users: zack.zacharias and deborah.demonstration
→ Bug #34922 → all class memberships have been removed

*** Test 6 (conflicts)

Benutzername,Vorname,Nachname,Klasse
Dennis1,Dennis,Demo,12a
Dennis1,Dennis Lars,Demonstration,10d
Daniela2,Daniela,Tester,13c
Daniela3,Daniela,Tester,13c
emil.langusername,Emil,Langusername,11b

→ Detected duplicate username (changed second line to username "Dennis2" and "modify" to "create")
→ created Dennis2 and Daniela3
→ created new class 10d

*** Test 7 (conflicts)

Vorname,Nachname,Klasse,Geburtstag
Dennis,Demo,12a,10.02.1997
Dennis Lars,Demonstration,10d,10.5.1997
Daniela,Tester,13c,10.02.1996
Daniela,Tester,12f,20.04.1996
emil.langusername,Emil,Langusername,11b,10.02.1997

→ detected username conflict ("daniela.tester")
→ detected invalid birthday ("11b")
→ fixed by ignoring last line and user username "daniela.tester2"
→ created 4 new users

*** Test 8 (conflicts)

Benutzername,Vorname,Nachname,Klasse,Passwort,E-Mail,Geburtstag
Dennis1,Dennis,Demo,12a,univention,dennis1@nstx.local,11.12.2009
Dennis1,Dennis Lars,Demonstration,10d,univention,dennis1@nstx.local,12.10.2009
Daniela2,Daniela,Tester,13c,univention,daniela2@nstx.local,11.12.2009
Daniela3,Daniela,Tester,13c,univention,daniela2@nstx.local,11.12.2009
emil.langusername,Emil,Langusername,11b,univention,daniela2@nstx.local,11.12.2009

→ all conflicts detected
→ changed mail address of emil to "emil@nstx.local" which is marked as unusual change (yellow)
→ 2 Fehler aufgetreten:

    Dennis1 konnte nicht verändert werden.
    Daniela2 konnte nicht verändert werden. 

20.05.14 17:34:10.588  MODULE      ( PROCESS ) : Going to modify CSVStudent Daniela2
[…]
20.05.14 17:34:10.659  MODULE      ( WARN    ) : Something went wrong. Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.6/univention/management/console/modules/schoolcsvimport/util.py", line 210, in commit
    self.modify(lo, validate=False)
  File "/usr/lib/pymodules/python2.6/ucsschool/lib/models/base.py", line 409, in modify
    success = self.modify_without_hooks(lo, validate, move_if_necessary)
  File "/usr/lib/pymodules/python2.6/ucsschool/lib/models/base.py", line 432, in modify_without_hooks
    self.do_modify(udm_obj, lo)
  File "/usr/lib/pymodules/python2.6/ucsschool/lib/models/user.py", line 203, in do_modify
    return super(User, self).do_modify(udm_obj, lo)
  File "/usr/lib/pymodules/python2.6/ucsschool/lib/models/base.py", line 458, in do_modify
    udm_obj.modify(ignore_license=1)
  File "/usr/lib/pymodules/python2.6/univention/admin/handlers/__init__.py", line 345, in modify
    return self._modify(modify_childs,ignore_license=ignore_license)
  File "/usr/lib/pymodules/python2.6/univention/admin/handlers/__init__.py", line 799, in _modify
    ml=self._ldap_modlist()
  File "/usr/lib/pymodules/python2.6/univention/admin/handlers/users/user.py", line 1996, in _ldap_modlist
    raise univention.admin.uexceptions.pwalreadyused
pwalreadyused


*** Test 9 (invalid header field, invalid lines, empty lines, comments, invalid mail address)

Benutzername,ungültig,Vorname,Nachname,Klasse,Passwort,E-Mail,Geburtstag
Dennis1,irgendwas,Dennis,Demo,12a,univention99A,dennis1@nstx.local,11.12.2009

Dennis1,somedata,Dennis Lars,Demonstration,10d,univention99A,dennis1@nstx.local,12.10.2009
# an irrelevant comment
Daniela2,ding,Daniela,Tester,13c,univention99A,daniela2@nstx.local,11.12.2009
[ broken line ... ]
Daniela3,dong,Daniela,Tester,13c,univention99A,daniela2@nstx.local,11.12.2009
emil.langusername,foo,Emil,Langusername,11b,univention99A,daniela2-nstx.local,11.12.2009

→ column "ungültig" is ignored
→ found conflicts in usernames
→ found conflicts in mail addresses
→ empty line has been ignored
→ the other two lines (starting with "#" and "[") were shown → it was possible to ignore them
→ the missing "@" in the mail address was detected → replaced with "emil@nstx.local"
→ no errors

*** Test 10 (no header, no username)

Dennis1,Dennis,Demo,12a
Daniela2,Daniela,Tester,13c
Daniela3,Daniela,Tester,13c
emil.langusername,Emil,Langusername,11b

→ manually specified the following columns: first name, last name, class (NOT username!)
→ The user "Daniela Tester" (line 2+3) was automatically mapped to user "Daniela2" and detected as conflict → after changing the second username to "Daniela3" the conflict was gone
→ no errors

*** Test 11 (latin1 encoding with DOS line breaks)

Benutzername,Vorname,Nachname,Klasse,Email
Dennis1,Dennis,Demo,12a,dennis1@nstx.local
Daniela2,Daniela,Tester,13c,daniela2@nstx.local
emil.langusername,Emil,Langusername,11b,wrong@address.com
lydia,Lydia,Lüttich,12a
jörg.lämmers,Jörg Walter,Lämmers von der Weile,13c

→ correctly parsed
→ no errors

*** Test 12 (latin1 encoding with MAC line breaks)

Benutzername,Vorname,Nachname,Klasse,Email
Dennis1,Dennis,Demo,12a,dennis1@nstx.local
Daniela2,Daniela,Tester,13c,daniela2@nstx.local
emil.langusername,Emil,Langusername,11b,wrong@address.com
lydia,Lydia,Lüttich,12a
jörg.lämmers,Jörg Walter,Lämmers von der Weile,13c

→ correctly parsed
→ no errors

*** Test 13 (import of 800 users)

6b;Heinz;Schreyer;2003-04-18
12b;Karl;Schmelz;2005-01-10
5d;Lara;Siefker ct. Schnieder;1980-10-29
6d;Manuel;Sickles;1976-10-18
9b;Nora;Stelzer;1993-10-01
5d;Felix;Street;1985-02-06
8d;Tom;Schlüter;1979-11-08
4e;Charlotte;Stein(s);1981-06-17
11c;Matthias;Aßmann;1997-10-22
2a;Nicole;Steinbrügge ct. Hagemeyer;1995-12-22
[…]

→ 13 duplicate usernames → manually fixed
→ import time: 3m14s

*** Test 14 (replace of 800 users with manual changes)

→ used data of test 13 and changed classes ("12a" → "K-12-a", ...)
→ all users were recognized with correct usernames / no conflicts
→ 13 test users existing from previous tests were marked for deletion → 7 of 13 have been marked for being ignored
→ import/replace time: 0m38s
→ no errors

*** Test 15 (import testfile 2 as teacher)

→ used testfile 4
→ all users were recognized as existing student users and shown as a conflict


QA successful
Comment 18 Sönke Schwardt-Krummrich univentionstaff 2014-06-12 09:19:21 CEST
UCS@school 3.2 R2 has been released:
http://docs.univention.de/release-notes-ucsschool-3.2R2-de.html

If this error occurs again, please use "Clone This Bug".