Bug 47447 - Support single source database, global recordUID with multiple partial import data sets
Support single source database, global recordUID with multiple partial import...
Status: CLOSED FIXED
Product: UCS@school
Classification: Unclassified
Component: Import scripts
UCS@school 4.3
Other Linux
: P5 normal (vote)
: UCS@school 4.3 v5
Assigned To: Daniel Tröder
Sönke Schwardt-Krummrich
:
Depends on: 47448
Blocks:
  Show dependency treegraph
 
Reported: 2018-08-02 15:19 CEST by Daniel Tröder
Modified: 2018-10-14 20:36 CEST (History)
2 users (show)

See Also:
What kind of report is it?: Feature Request
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments
Some CSV files with testdata (532 bytes, application/x-compressed-tar)
2018-09-10 22:25 CEST, Sönke Schwardt-Krummrich
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Tröder univentionstaff 2018-08-02 15:19:13 CEST
Implement a possibility to run imports through the command line and the HTTP-API-import UI, that enable the following scenario:

Requirements:

* A single source database exists that knows all users and has globally unique recordUIDs them.
* OU spanning user accounts are used (a user can be member of multiple schools).
* The source database exports separate CSV files per school and user type.
* Each school imports its users separately at a time and order of their choosing.
* As imports are done in random order, it is possible that to move a user from one school to another,it is first removed in one school and imported at the other school at a later time. The user account must not be deleting in the meantime.

Implementation:

* When importing, the users of the same source database (same sourceUID) that are not in the CSV file should not be deleted:
  * When searching for existing users to delete (because they are missing in the CSV file), only those users that are part of the importing school should be considered.
  * Users that normally would be deleted (or deactivated), will be immediately deactivated and moved to a temporary OU (henceforth called "limbo" OU/school) instead.
* If a user is to be created in the importing school, a search for a user with its recordUID is done. If it exists in any school of the domain (including the limbo school), the school being imported is added to its ``schools`` attribute. If the user was in the limbo school, it's removed from there and thus moved from it to the school being imported.
Comment 1 Daniel Tröder univentionstaff 2018-08-02 17:10:00 CEST
To use this, just add /usr/share/ucs-school-import/configs/user_import_sisopi.json to your import configuration.

A ucs-test was added: 239_import-users_sisopi.

[4.3] fea32525d Bug #47447: refactor detect_users_to_delete()
[4.3] 1c8a671a1 Bug #47447: refactor determine_add_modify_action()
[4.3] 2c1574ffb Bug #47447: implement single source database, partial import user import scenario
[4.3] 9f02fdd36 Bug #47447: PEP8 fixes
[4.3] 24f7388de Bug #47447: remove unused code
[4.3] a05218ad3 Bug #47447: improve documentation
[4.3] ddeef0d8f Bug #47447: add ucs-test for single source database, partial import user import scenario
[4.3] 218f92760 Bug #47447 Bug #47448: changelog
[4.3] e0d7a329b Bug #47447 Bug #47448: Merge branch 'dtroeder/SiSoMi.scenario' into 4.3
[4.3] d02743564 Bug #47447 Bug #47448: advisory

ucs-school-import (16.0.2-30)
ucs-test-ucsschool (5.0.2-75)

The internal API documentation is online: https://billy.knut.univention.de/~dtroeder/http-api-doc/python/ucsschool.importer.mass_import.html#module-ucsschool.importer.mass_import.sisopi_user_import
Comment 2 Sönke Schwardt-Krummrich univentionstaff 2018-08-31 17:03:26 CEST
Requirements:
- same sourceUID for all schools (per user type)
- "school" is always set within the configuration
- deletion_grace_period::deactivation=0

REOPEN:
- deletion_grace_period::deactivation has to be always 0 because the code uses deactivate_now() during each move to limbo ou ==> no effect if other value is set ==> raise InitialisationError
- "Enduser documentation"/internal documentation is missing.

To be done:
- code review for ucs-test script
- manual tests
Comment 3 Daniel Tröder univentionstaff 2018-09-04 13:09:34 CEST
[4.3] 42fe58f40 Bug #47447: separate module docstring from license notice
[4.3] 80dabdb3c Bug #47447: deletion_grace_period::deactivation == 0 is a configuration error
[4.3] 20fe3b6f1 Bug #47447: add section to internal documentation
[4.3] 6f59227ea Bug #47447: build hooks graph in buildsystem, add section about hooks to internal documentation

(In reply to Sönke Schwardt-Krummrich from comment #2)
> REOPEN:
> - deletion_grace_period::deactivation has to be always 0 because the code
> uses deactivate_now() during each move to limbo ou ==> no effect if other
> value is set ==> raise InitialisationError
80dabdb3c

> - "Enduser documentation"/internal documentation is missing.
6f59227ea

The documentation for the Single source, partial import (SiSoPi) scenario can be found here: https://billy.knut.univention.de/~dtroeder/http-api-doc/sisopi.html
Comment 4 Sönke Schwardt-Krummrich univentionstaff 2018-09-07 15:00:14 CEST
> (In reply to Sönke Schwardt-Krummrich from comment #2)
> > REOPEN:
> > - deletion_grace_period::deactivation has to be always 0 because the code
> > uses deactivate_now() during each move to limbo ou ==> no effect if other
> > value is set ==> raise InitialisationError
> 80dabdb3c
→ OK

> > - "Enduser documentation"/internal documentation is missing.
> 6f59227ea
→ OK

> The documentation for the Single source, partial import (SiSoPi) scenario
> can be found here:
> https://billy.knut.univention.de/~dtroeder/http-api-doc/sisopi.html
→ OK

REOPEN: manual tests
OK: ucs-test script

The HTTP-API always sets a per school specific sourceUID. Therefore the is no global sourceUID possible, and therefore SiSoPI does not work with the HTTP API (at the moment).
Comment 5 Daniel Tröder univentionstaff 2018-09-07 15:51:50 CEST
* A new UCR variable ucsschool/import/http_api/set_source_uid (default true) can be disabled to retain the source_uid form the configuration file in a HTTP-API import. Before it always changed it to "<OU>-<role>", effectively disabling the SiSoPi scenario.

* The source_uid field is now read-only in the UserImportJob resource.

* A bug in the ucsschool.lib didn't create the "schools" attribute in a User object, when created from an UDM object. That resulted in repeated tries to add a school to a user (because User.old_user.schools contained only the primary school).

[4.3] 9873b68ad Bug #47447: set 'schools' property when creating User from UDM object
[4.3] 43d36d930 Bug #47447: make source_uid read-only field
[4.3] f7184f509 Bug #47447: don't change source_uid in HTTP-API import in SiSoPi scenario
[4.3] 30ca12b5c Bug #47447: advisories

ucs-school-lib (11.0.1-22)
ucs-school-import (16.0.2-44)
Comment 6 Sönke Schwardt-Krummrich univentionstaff 2018-09-10 10:00:27 CEST
Looks mostly very good but I found one issue:
If the user is moved to the limbo-school (here: "graveyard"), the user is removed from schueler-$OU, Domain Users $OU but remains in all class groups:

 dn: cn=schueler-schule2,cn=groups,ou=schule2,dc=nstx,dc=local
-uniqueMember: uid=JohannWolfga5,cn=schueler,cn=users,ou=schule2,dc=nstx,dc=local
-memberUid: JohannWolfga5

 dn: cn=schule2-Neueinstellung,cn=klassen,cn=schueler,cn=groups,ou=schule2,dc=nstx,dc=local
-uniqueMember: uid=JohannWolfga5,cn=schueler,cn=users,ou=schule2,dc=nstx,dc=local
+uniqueMember: uid=JohannWolfga5,cn=schueler,cn=users,ou=graveyard,dc=nstx,dc=local

 dn: cn=Domain Users schule2,cn=groups,ou=schule2,dc=nstx,dc=local
-uniqueMember: uid=JohannWolfga5,cn=schueler,cn=users,ou=schule2,dc=nstx,dc=local
-memberUid: JohannWolfga5

 dn: cn=schule3-Poeten,cn=klassen,cn=schueler,cn=groups,ou=schule3,dc=nstx,dc=local
-uniqueMember: uid=JohannWolfga5,cn=schueler,cn=users,ou=schule2,dc=nstx,dc=local
+uniqueMember: uid=JohannWolfga5,cn=schueler,cn=users,ou=graveyard,dc=nstx,dc=local
Comment 7 Daniel Tröder univentionstaff 2018-09-10 14:38:37 CEST
Fixed and added a test for it.

[4.3] 708b1ddf4 Bug #47447: remove user from school classes when moving to limbo_ou
[4.3] 95a2e67f3 Bug #47447: advisory update

ucs-school-import (16.0.2-47)
Comment 8 Sönke Schwardt-Krummrich univentionstaff 2018-09-10 22:25:17 CEST
Created attachment 9669 [details]
Some CSV files with testdata

I tested manually with the attached testdata. For testing, you have to set up the environment:
1) ucr set ucsschool/import/http_api/set_source_uid=no
2) # cat /var/lib/ucs-school-import/configs/user_import.json 
{
        "classes": {
                "reader": "ucsschool.importer.reader.http_api_csv_reader.HttpApiCsvReader",
                "user_importer": "ucsschool.importer.mass_import.sisopi_user_import.SingleSourcePartialUserImport"

        },
        "configuration_checks": ["defaults", "sisopi"],
        "deletion_grace_period": {
               "deactivation": 0,
               "deletion": 90
        },
        "limbo_ou": "graveyard",
        "csv": {
                "mapping": {
                        "recordUID": "record_uid",
                        "firstname": "firstname",
                        "lastname": "lastname",
                        "school_classes": "school_classes"
                }
        },
        "scheme": {
                "username": {
                        "default": "<firstname>[ALWAYSCOUNTER]"
                }
        },
        "verbose": true,
        "sourceUID": "apistudents"
}
3) create OUs "schule1", "schule2", "schule3", "graveyard" and create a teacher in each school
4) give each created teacher the permission to use the new UMC module "Benutzerimport" (via HTTP API)
5) import the attached CSV files in correct order:
   school1a.csv  →  import in "schule1" with teacher1  (a → first import)
   school2b.csv  →  import in "schule2" with leacher2  (b → second import)
   school2c.csv  →  import in "schule2" with leacher2  (c → third import)
   school3d.csv  →  import in "schule3" with leacher3  (d → fourth import)
   and so on
Comment 9 Sönke Schwardt-Krummrich univentionstaff 2018-09-10 22:34:13 CEST
The manual import looks ok now → VERIFIED

OK: code change
OK: manual tests (see last post)
OK: ucs-test
OK: advisory
Comment 10 Sönke Schwardt-Krummrich univentionstaff 2018-09-11 11:34:19 CEST
UCS@school 4.3 v5 has been released.

https://docs.software-univention.de/changelog-ucsschool-4.3v5-de.html

If this error occurs again, please clone this bug.