Univention Bugzilla – Bug 52920
ASM: Inconsistent data if LDAP filters are used
Last modified: 2021-05-05 17:11:38 CEST
In the file classes.py the classes.csv is created, which contains a list of all classes and workgroups including their teachers. Each teacher is listed in a separate CSV column (max. 15). But there is currently no check, if a teacher is not excluded by one of the LDAP filters configured via UCR. So if a teacher is referenced in a class that does not appear in the staff.csv file due to LDAP filter exclusion, the ASM will have a problem. Instead of simply taking the list of teachers from the "SchoolClass" object, this should actually be filtered once against the output of the "get_staff()" method, which handles the LDAP filters.
Implemented changes in [twenzel/asm/52920_inconsistent_ldap_filters] e134702 Bug #52920: filter out excluded staff I moved the functionality of get_staff to a class method in AsmStaff to use the code in csv_file.py & classes.py. I split the 11_ldap_filters test to make it more readable (you will see what I mean in the code). There is also a test for (!(uid=user_name)) to prove this bug is fixed.
The general idea looks good. I have the following proposals for improvement: - remove spaces from your commit - split into different test cases - dont refetch User objects in classes.py, use already existing Users from get_filtered_staff function
Thanks for the QA! I implemented the requested changes in [twenzel/asm/52920_inconsistent_ldap_filters] 0ed35bd fixup! Bug #52920: filter out excluded staff As discussed, - I also added the `.py` for all tests in asm - added a test for get_filtered_staff tested in single + muli-server env asm version 2.2.4 apple-school-manager_20210412123152 has been created/ prepared.
As discussed, changes are pushed to 4.4 [4.4] ac8e5c1 Bug #52920: add changelog [4.4] a58b5e4 Bug #52920: filter out excluded staff
The solution was already tested & works but is not very efficient. In [twenzel/asm/52920_inconsistent_ldap_filters] 9b6c497 Bug #52920: cache teachers in school I implemented a suggestion, but I'm not 100% sure :)
Fixed import [twenzel/asm/52920_inconsistent_ldap_filters] 9b6c497 Bug #52920: cache teachers in school The solution seems to be ok, I tested this as follows: created 500 teachers for DEMOSCHOOL & unset the ucr-vs → this way all teachers are loaded timed the refactored function teachers = get_filtered_staff(lo, logger, "DEMOSCHOOL") Before the fix it takes 1-1.3 s to load the teachers after the fix it takes around 1.3 s the first time the function is executed.
* _teachers_in_school in classes.py is unused * revert indent of config_files-value in csv_file.py:210 * Please fix those old errors: ./modules/univention/asm/csv/zip_file.py:51:2: F401 'typing.List' imported but unused ./modules/univention/asm/csv/csv_file.py:241:3: F841 local variable 'exc' is assigned to but never used ./modules/univention/asm/models/base.py:129:3: F821 undefined name 'ucsschool' ./modules/univention/asm/models/classes.py:42:1: F401 'ucsschool.lib.models.User' imported but unused ./93_apple-school-manager/10_csv_file.py:26:2: F401 'typing.List' imported but unused ./93_apple-school-manager/20_zip_file.py:26:2: F401 'typing.List' imported but unused ./93_apple-school-manager/02_model_course.py:12:1: F401 'univention.config_registry.ConfigRegistry' imported but unused * Code looks good, but I cannot test this. Please rebase the branch twenzel/asm/52920_inconsistent_ldap_filters upon 4.4. The package in the branch is 3 version behind 4.4! I don't know what to QA.
Thanks for the QA so far! Sorry for this mess, I implemented your remarks and rebased on 4.4 [twenzel/asm/52920_inconsistent_ldap_filters] 6e2b0ef Bug #52920: qa remarks [twenzel/asm/52920_inconsistent_ldap_filters] 3c689bc Bug #52920: cache teachers in school
The package "python-backports.functools-lru-cache" is in unmaintained. Please upload it into the apps repository.
The package "python-backports.functools-lru-cache" has been moved to maintained and will be released next Wednesday as UCS 4.4-8 errata.
python-backports.functools-lru-cache has been added to maintained. It will be available with the next errata [4.4-8] cf916db415 Bug #52920: add package to maintaned for apple-school-manager teachers are now cached to speed up the process. [4.4] b2fed4c Bug #52920: add changelog [4.4] 59081d6 Bug #52920: cache teachers in school Package: univention-apple-school-manager-connector Version: 2.0.0-10A~4.4.0.202104291544 Branch: ucs_4.4-0 Scope: univention-asm app has been uploaded to test appcenter (2.2.4)
OK: automatic tests OK: changelog OK: test-appcenter OK: manual test: root@m20:~# univention-ldapsearch -LLL ucsschoolRole=teacher:school:Gym21 | egrep 'dn:|objectClass: ucsschool' dn: uid=gymadmin,cn=lehrer,cn=users,ou=Gym21,dc=uni,dc=dtr objectClass: ucsschoolAdministrator objectClass: ucsschoolTeacher objectClass: ucsschoolType dn: uid=Lehr21,cn=lehrer,cn=users,ou=Gym21,dc=uni,dc=dtr objectClass: ucsschoolTeacher objectClass: ucsschoolType root@m20:~# ucr set 'asm/ldap_filter/staff=(!(objectClass=ucsschoolAdministrator))' root@m20:~# ucr set asm/username=foo asm/store_zip=true root@m20:~# echo univention > /etc/asm.secret root@m20:~# asm-upload Creating ZIP file in /var/lib/asm/asm_2021-04-30T09:48:52.627799.zip... Creating CSV files... Writing 3 objects to students.csv... Writing 12 objects to locations.csv... Writing 2 objects to staff.csv... No handlers could be found for logger "ucsschool.lib.models.group" Writing 3 objects to rosters.csv... Writing 26 objects to classes.csv... Writing 26 objects to courses.csv... Finished creating CSV files. Finished creating ZIP file. Uploading ZIP file to upload.appleschoolcontent.com... SFTP upload failed: Authentication failed. root@m20:~# cd /tmp/ root@m20:/tmp# unzip /var/lib/asm/asm_2021-04-30T09\:48\:52.627799.zip root@m20:/tmp# grep gymadmin *.csv || echo "no gymadmin in csv" no gymadmin in csv
(In reply to Tobias Wenzel from comment #12) > python-backports.functools-lru-cache has been added to maintained. > It will be available with the next errata Activating the errata test scope shows it ready for release: root@m20:~# apt-cache policy python-backports.functools-lru-cache python-backports.functools-lru-cache: Installiert: 1.3-1 Installationskandidat: 1.3-1 Versionstabelle: *** 1.3-1 500 500 http://updates-test.software-univention.de/4.3/unmaintained 4.3-0/all/ Packages 500 http://updates-test.software-univention.de/4.4/unmaintained/component 4.4-8-errata-test/all/ Packages 100 /var/lib/dpkg/status
(In reply to Daniel Tröder from comment #14) > Activating the errata test scope shows it ready for release: > > http://updates-test.software-univention.de/4.4/unmaintained/component > 4.4-8-errata-test/all/ Packages No it is not, the package is shown in unmaintained!
python-backports.functools-lru-cache is going to be force-announced to maintained -> back to resolved
OK: package will be released: ------------------------------------------------------------------------------- root@m20:~# apt update [..] root@m20:~# apt-cache policy python-backports.functools-lru-cache python-backports.functools-lru-cache: Installiert: 1.3-1 Installationskandidat: 1.3-1 Versionstabelle: *** 1.3-1 500 500 http://updates-test.software-univention.de/4.3/unmaintained 4.3-0/all/ Packages 500 http://updates-test.software-univention.de/4.4/maintained/component 4.4-8-errata-test/all/ Packages 100 /var/lib/dpkg/status
<https://errata.software-univention.de/#/?erratum=4.4x969>