Univention Bugzilla – Bug 47447
Support single source database, global recordUID with multiple partial import data sets
Last modified: 2018-10-14 20:36:21 CEST
Implement a possibility to run imports through the command line and the HTTP-API-import UI, that enable the following scenario: Requirements: * A single source database exists that knows all users and has globally unique recordUIDs them. * OU spanning user accounts are used (a user can be member of multiple schools). * The source database exports separate CSV files per school and user type. * Each school imports its users separately at a time and order of their choosing. * As imports are done in random order, it is possible that to move a user from one school to another,it is first removed in one school and imported at the other school at a later time. The user account must not be deleting in the meantime. Implementation: * When importing, the users of the same source database (same sourceUID) that are not in the CSV file should not be deleted: * When searching for existing users to delete (because they are missing in the CSV file), only those users that are part of the importing school should be considered. * Users that normally would be deleted (or deactivated), will be immediately deactivated and moved to a temporary OU (henceforth called "limbo" OU/school) instead. * If a user is to be created in the importing school, a search for a user with its recordUID is done. If it exists in any school of the domain (including the limbo school), the school being imported is added to its ``schools`` attribute. If the user was in the limbo school, it's removed from there and thus moved from it to the school being imported.
To use this, just add /usr/share/ucs-school-import/configs/user_import_sisopi.json to your import configuration. A ucs-test was added: 239_import-users_sisopi. [4.3] fea32525d Bug #47447: refactor detect_users_to_delete() [4.3] 1c8a671a1 Bug #47447: refactor determine_add_modify_action() [4.3] 2c1574ffb Bug #47447: implement single source database, partial import user import scenario [4.3] 9f02fdd36 Bug #47447: PEP8 fixes [4.3] 24f7388de Bug #47447: remove unused code [4.3] a05218ad3 Bug #47447: improve documentation [4.3] ddeef0d8f Bug #47447: add ucs-test for single source database, partial import user import scenario [4.3] 218f92760 Bug #47447 Bug #47448: changelog [4.3] e0d7a329b Bug #47447 Bug #47448: Merge branch 'dtroeder/SiSoMi.scenario' into 4.3 [4.3] d02743564 Bug #47447 Bug #47448: advisory ucs-school-import (16.0.2-30) ucs-test-ucsschool (5.0.2-75) The internal API documentation is online: https://billy.knut.univention.de/~dtroeder/http-api-doc/python/ucsschool.importer.mass_import.html#module-ucsschool.importer.mass_import.sisopi_user_import
Requirements: - same sourceUID for all schools (per user type) - "school" is always set within the configuration - deletion_grace_period::deactivation=0 REOPEN: - deletion_grace_period::deactivation has to be always 0 because the code uses deactivate_now() during each move to limbo ou ==> no effect if other value is set ==> raise InitialisationError - "Enduser documentation"/internal documentation is missing. To be done: - code review for ucs-test script - manual tests
[4.3] 42fe58f40 Bug #47447: separate module docstring from license notice [4.3] 80dabdb3c Bug #47447: deletion_grace_period::deactivation == 0 is a configuration error [4.3] 20fe3b6f1 Bug #47447: add section to internal documentation [4.3] 6f59227ea Bug #47447: build hooks graph in buildsystem, add section about hooks to internal documentation (In reply to Sönke Schwardt-Krummrich from comment #2) > REOPEN: > - deletion_grace_period::deactivation has to be always 0 because the code > uses deactivate_now() during each move to limbo ou ==> no effect if other > value is set ==> raise InitialisationError 80dabdb3c > - "Enduser documentation"/internal documentation is missing. 6f59227ea The documentation for the Single source, partial import (SiSoPi) scenario can be found here: https://billy.knut.univention.de/~dtroeder/http-api-doc/sisopi.html
> (In reply to Sönke Schwardt-Krummrich from comment #2) > > REOPEN: > > - deletion_grace_period::deactivation has to be always 0 because the code > > uses deactivate_now() during each move to limbo ou ==> no effect if other > > value is set ==> raise InitialisationError > 80dabdb3c → OK > > - "Enduser documentation"/internal documentation is missing. > 6f59227ea → OK > The documentation for the Single source, partial import (SiSoPi) scenario > can be found here: > https://billy.knut.univention.de/~dtroeder/http-api-doc/sisopi.html → OK REOPEN: manual tests OK: ucs-test script The HTTP-API always sets a per school specific sourceUID. Therefore the is no global sourceUID possible, and therefore SiSoPI does not work with the HTTP API (at the moment).
* A new UCR variable ucsschool/import/http_api/set_source_uid (default true) can be disabled to retain the source_uid form the configuration file in a HTTP-API import. Before it always changed it to "<OU>-<role>", effectively disabling the SiSoPi scenario. * The source_uid field is now read-only in the UserImportJob resource. * A bug in the ucsschool.lib didn't create the "schools" attribute in a User object, when created from an UDM object. That resulted in repeated tries to add a school to a user (because User.old_user.schools contained only the primary school). [4.3] 9873b68ad Bug #47447: set 'schools' property when creating User from UDM object [4.3] 43d36d930 Bug #47447: make source_uid read-only field [4.3] f7184f509 Bug #47447: don't change source_uid in HTTP-API import in SiSoPi scenario [4.3] 30ca12b5c Bug #47447: advisories ucs-school-lib (11.0.1-22) ucs-school-import (16.0.2-44)
Looks mostly very good but I found one issue: If the user is moved to the limbo-school (here: "graveyard"), the user is removed from schueler-$OU, Domain Users $OU but remains in all class groups: dn: cn=schueler-schule2,cn=groups,ou=schule2,dc=nstx,dc=local -uniqueMember: uid=JohannWolfga5,cn=schueler,cn=users,ou=schule2,dc=nstx,dc=local -memberUid: JohannWolfga5 dn: cn=schule2-Neueinstellung,cn=klassen,cn=schueler,cn=groups,ou=schule2,dc=nstx,dc=local -uniqueMember: uid=JohannWolfga5,cn=schueler,cn=users,ou=schule2,dc=nstx,dc=local +uniqueMember: uid=JohannWolfga5,cn=schueler,cn=users,ou=graveyard,dc=nstx,dc=local dn: cn=Domain Users schule2,cn=groups,ou=schule2,dc=nstx,dc=local -uniqueMember: uid=JohannWolfga5,cn=schueler,cn=users,ou=schule2,dc=nstx,dc=local -memberUid: JohannWolfga5 dn: cn=schule3-Poeten,cn=klassen,cn=schueler,cn=groups,ou=schule3,dc=nstx,dc=local -uniqueMember: uid=JohannWolfga5,cn=schueler,cn=users,ou=schule2,dc=nstx,dc=local +uniqueMember: uid=JohannWolfga5,cn=schueler,cn=users,ou=graveyard,dc=nstx,dc=local
Fixed and added a test for it. [4.3] 708b1ddf4 Bug #47447: remove user from school classes when moving to limbo_ou [4.3] 95a2e67f3 Bug #47447: advisory update ucs-school-import (16.0.2-47)
Created attachment 9669 [details] Some CSV files with testdata I tested manually with the attached testdata. For testing, you have to set up the environment: 1) ucr set ucsschool/import/http_api/set_source_uid=no 2) # cat /var/lib/ucs-school-import/configs/user_import.json { "classes": { "reader": "ucsschool.importer.reader.http_api_csv_reader.HttpApiCsvReader", "user_importer": "ucsschool.importer.mass_import.sisopi_user_import.SingleSourcePartialUserImport" }, "configuration_checks": ["defaults", "sisopi"], "deletion_grace_period": { "deactivation": 0, "deletion": 90 }, "limbo_ou": "graveyard", "csv": { "mapping": { "recordUID": "record_uid", "firstname": "firstname", "lastname": "lastname", "school_classes": "school_classes" } }, "scheme": { "username": { "default": "<firstname>[ALWAYSCOUNTER]" } }, "verbose": true, "sourceUID": "apistudents" } 3) create OUs "schule1", "schule2", "schule3", "graveyard" and create a teacher in each school 4) give each created teacher the permission to use the new UMC module "Benutzerimport" (via HTTP API) 5) import the attached CSV files in correct order: school1a.csv → import in "schule1" with teacher1 (a → first import) school2b.csv → import in "schule2" with leacher2 (b → second import) school2c.csv → import in "schule2" with leacher2 (c → third import) school3d.csv → import in "schule3" with leacher3 (d → fourth import) and so on
The manual import looks ok now → VERIFIED OK: code change OK: manual tests (see last post) OK: ucs-test OK: advisory
UCS@school 4.3 v5 has been released. https://docs.software-univention.de/changelog-ucsschool-4.3v5-de.html If this error occurs again, please clone this bug.