Univention Bugzilla – Bug 53841
Import summary of dry-run contains unmodified LDAP data instead of values from input data
Last modified: 2021-10-20 15:03:53 CEST
Created attachment 10829 [details] Dry-run job folder A customer had a problem with the recordUID of their users. They realized it and wanted to fall back by using the import data of the import before. They checked the summary of the dry-run, but the summary did not show the corrected configuration as it would look like, if the dry run is no dry run. The summary must show how it would look if the import will be done, not the unchanged configuration. Maybe with the remark Dry-Run. As the customer import contains real user data, I verified the behavior. You will find job zips attached from my test environment. It is really easy to reproduce; just do some changes you will find in the summary and do a dry-run; the change will not be shown.
The file user_import_summary.csv shows the configuration as there was no import. It must (from customers and my point of view) show how it would look like, if the import would be no dry run. F.e. User Melina Röhling; should get the recordUID 618 (see 1632722167-Schueler_Import_2020-01_Schule2-students_kurz.csv last entry) the file user_import_summary.csv says the recordUID is 121, what means unchanged. The summary file should present how it looks like, if the changes have been done; in this way the summary confuses the user about the result.
Something *partly* messed up the order in the file user_import_summary.csv. All columns except "line" and "record_uid" are in revered order. The first user from the input-csv file is the last user in the summary file. Only the columns "line" and "record_uid" are in the correct order. I cannot reproduce this behavior. I have looked into my test-VMs and a customer VM and I cannot find any other import that has produced such a strange user_import_summary.csv file.
No Daniel! That was my way to repruduce the customer problem. I changed the order of rechordUID in the import file; the customers school adminitration tool srambled the identifier that is/was used as recordUID. Perhaps we should talk about the problem?
Need more info without a question and no answer on my offer to talk about the problem? I don't know how to explain more detail, but I will try again: If you do a dry run the user_import_summary.csv file do not show what would be the result of the import. It shows the actual and unchanged status of users without a import process. This file should show how the users would look like if the new data were processed. It must be possible to see what would happen if you do the next step, the real import. The presentation of the status quo makes no sense. Why? The filename user_import_summary.csv suggests that you will find the _summary_ of this job; no matter if dry-run or real import. The customer checked if the job would repair the problem, but the file contained unchanged users, so they skipped the healing real import. If you need more info please contact me direct.
The import *cannot* be used to change record_uid entries. If that is what the customer tried, then the summary is correct: nothing changes.
I have made an import and changed the firstname and lastname of users. The result is in the summary file as expected: ----------------------------------------------- root@m70:~# /usr/share/ucs-school-import/scripts/ucs-school-testuser-import -n --httpapi --students 3 --classes 1 --verbose DEMOSCHOOL root@m70:~# vi 'test_users_2021-09-28_13:59:50.csv' # add "record_uid" column root@m70:~# cp /usr/share/ucs-school-import/configs/ucs-school-testuser-http-import.json . root@m70:~# vi ucs-school-testuser-http-import.json # add "record_uid"->"record_uid" to mapping root@m70:~# cat 'test_users_2021-09-28_13:59:50.csv' "Schule","Vorname","Nachname","Klassen","Beschreibung","Telefon","EMail","record_uid" "DEMOSCHOOL","Inula","Wengert","1a","A student.","+10-796-342190","","111" "DEMOSCHOOL","Anja","Sudendorf","1a","A student.","+17-538-612448","","222" "DEMOSCHOOL","Ainers","Kraimer","1a","A student.","+26-127-748992","","333" root@m70:~# /usr/share/ucs-school-import/scripts/ucs-school-user-import --school DEMOSCHOOL --user_role student --source_uid demoschool-student --conffile ./ucs-school-testuser-http-import.json --infile 'test_users_2021-09-28_13:59:50.csv' ------ User import statistics ------ Created ImportStudent: 3 ['inula.wengert', 'anja.sudendorf', 'ainers.kraimer'] ------ Writing user import summary to /var/lib/ucs-school-import/summary/2021/09/user_import_summary_2021-09-28_14:02:03.csv... ------ root@m70:~# cat /var/lib/ucs-school-import/summary/2021/09/user_import_summary_2021-09-28_14\:02\:03.csv "line","success","error","action","role","username","schools","firstname","lastname","birthday","email","disabled","classes","source_uid","record_uid","error_msg" "2","1","0","A","student","inula.wengert","DEMOSCHOOL","Inula","Wengert","","","0","DEMOSCHOOL-1a","demoschool-student","111","" "3","1","0","A","student","anja.sudendorf","DEMOSCHOOL","Anja","Sudendorf","","","0","DEMOSCHOOL-1a","demoschool-student","222","" "4","1","0","A","student","ainers.kraimer","DEMOSCHOOL","Ainers","Kraimer","","","0","DEMOSCHOOL-1a","demoschool-student","333","" root@m70:~# sed -e 's/Inula/Daniel/g' -e 's/Kraimer/Tröder/g' 'test_users_2021-09-28_13:59:50.csv' > 'test_users_2021-09-28_13:59:50_changed_names.csv' root@m70:~# diff 'test_users_2021-09-28_13:59:50.csv' 'test_users_2021-09-28_13:59:50_changed_names.csv' 2c2 < "DEMOSCHOOL","Inula","Wengert","1a","A student.","+10-796-342190","","111" --- > "DEMOSCHOOL","Daniel","Wengert","1a","A student.","+10-796-342190","","111" 4c4 < "DEMOSCHOOL","Ainers","Kraimer","1a","A student.","+26-127-748992","","333" --- > "DEMOSCHOOL","Ainers","Tröder","1a","A student.","+26-127-748992","","333" root@m70:~# /usr/share/ucs-school-import/scripts/ucs-school-user-import --school DEMOSCHOOL --user_role student --source_uid demoschool-student --conffile ./ucs-school-testuser-http-import.json --infile 'test_users_2021-09-28_13:59:50_changed_names.csv' ------ Created 0 users, modified 3 users. ------ Modified ImportStudent: 3 ['inula.wengert', 'anja.sudendorf', 'ainers.kraimer'] ------ Writing user import summary to /var/lib/ucs-school-import/summary/2021/09/user_import_summary_2021-09-28_14:03:54.csv... ------ root@m70:~# cat /var/lib/ucs-school-import/summary/2021/09/user_import_summary_2021-09-28_14\:03\:54.csv "line","success","error","action","role","username","schools","firstname","lastname","birthday","email","disabled","classes","source_uid","record_uid","error_msg" "2","1","0","M","student","inula.wengert","DEMOSCHOOL","Daniel","Wengert","","","0","DEMOSCHOOL-1a","demoschool-student","111","" "3","1","0","M","student","anja.sudendorf","DEMOSCHOOL","Anja","Sudendorf","","","0","DEMOSCHOOL-1a","demoschool-student","222","" "4","1","0","M","student","ainers.kraimer","DEMOSCHOOL","Ainers","Tröder","","","0","DEMOSCHOOL-1a","demoschool-student","333",""
My example above has an error: it does not use the "dry-run", but executes a real import. In case of a real import the summary is correct, but in case of a dry-run it's not. The problem is, that before executing post_modify hooks, the user is reloaded from LDAP. That is done, because UDM and UDM hooks can modify the user object without the UCS@school import knowing about it, and post_modify hooks should work on updated objects. In case of a dry-run the modifications that had been done on behalf of the input data, are reverted by the reload. Thus the summary contains only the user data from LDAP. Modify ImportUser.call_hooks() such that in case of a dry-run it does not reload the user from LDAP.
Daniel I thought some more about the problem and the solution you explained. The summary will also not be correct despite the correction on a dryrun when hooks are applied; at least the effects of the hooks will be missing. Since the customer has the issue with the bug in their school management software, they are asking if they can get a patch (I think without the issue I mentioned above).
(In reply to Dirk Schnick from comment #8) > Daniel I thought some more about the problem and the solution you explained. > The summary will also not be correct despite the correction on a dryrun when > hooks are applied; at least the effects of the hooks will be missing. The hooks are executed in dry-run, when the hooks set "supports_dry_run=True". > Since the customer has the issue with the bug in their school management > software, they are asking if they can get a patch (I think without the issue > I mentioned above). I fixed the bug, it just needs QA and release.
Fixed in branch dtroeder/53841_no_user_reload_in_dryrun: 8ead34d1f Bug #53841: no not reset user data to state in LDAP during dry-run 47f9c0d04 Bug #53841: improve comments 609fe981f Bug #53841: add test for correct values in summary file after a dry-run
For QA: How to reproduce ------------------------ 1. Create test user data. # /usr/share/ucs-school-import/scripts/ucs-school-testuser-import -n --httpapi --students 3 --classes 1 DEMOSCHOOL 2. Modify the csv data to add a "record_uid" column, so importing the same file again will work. # vi test_users_2021-*.csv 3. Copy and edit ucs-school-testuser-http-import.json so the "record_uid" column will be used in the imports. # cp /usr/share/ucs-school-import/configs/ucs-school-testuser-http-import.json . # vi ucs-school-testuser-http-import.json 4. Import for real. # /usr/share/ucs-school-import/scripts/ucs-school-user-import --school DEMOSCHOOL --user_role student --source_uid demoschool-student --conffile ./ucs-school-testuser-http-import.json --infile test_users_2021-*.csv 5. Edit CSV file to modify firstname and lastname. # vi test_users_2021-*.csv 6. Import with *dry-run*. # /usr/share/ucs-school-import/scripts/ucs-school-user-import -n --school DEMOSCHOOL --user_role student --source_uid demoschool-student --conffile ./ucs-school-testuser-http-import.json --infile test_users_2021-*.csv 7. Look at user_import_summary_....csv and see that the content is that of the original import - the changes to firstname and lastname are not in there. But in the log(file) the changes can be seen - they are just not stored in the summary.csv.
The git branch "dtroeder/53841_5.0_no_user_reload_in_dryrun" contains the commits for 5.0.
All OK Reopen for merge & build in 4.4 & 5.0 QA in 4.4 before patch -> first + last name are not changed in the user_import_summary -> visible in log -> test fails after patch -> first + last name are changed in the user_import_summary -> (still) visible in log -> test succeeds OK changelog OK code QA in 5.0 before patch -> first + last name are not changed in the user_import_summary -> visible in log after patch -> first + last name are changed in the user_import_summary -> (still) visible in log OK changelog OK code
Merged to 4.4 and 5.0, and built: [4.4] 130f4e3a4 Bug #53841: no not reset user data to state in LDAP during dry-run [4.4] 75df117b6 Bug #53841: improve comments [4.4] 6d33ab6cd Bug #53841: add test for correct values in summary file after a dry-run [4.4] 71c3d08f1 Bug #53841: Merge branch 'dtroeder/53841_no_user_reload_in_dryrun' into 4.4 [4.4] 02283dc1f Bug #53841: changelogs, advisory ucs-school-import (17.0.73) ucs-test-ucsschool (6.0.250) ------------------------------------------------------------------ (Here I remembered to squash before merging...) [5.0] e1ee8d32c Bug #53841: do not reset user data to state in LDAP during dry-run ucs-school-import (18.0.4) ucs-test-ucsschool (7.3.5)
[5.0 8d6dfc39e] Bug #53841: execute test in Python 3 ucs-test-ucsschool (7.3.6)
QA: All OK -> Verify - Jenkins happy in 5.0 - Jenkins happy (test passes) in 4.4 - changelogs OK - yaml verion OK - merges OK
Errata updates for UCS@school 4.4 v9 have been released. https://docs.software-univention.de/changelog-ucsschool-4.4v9-de.html If this error occurs again, please clone this bug.